2022-11-23T01:30:12.1518221Z Requested labels: linux.rocm.gpu 2022-11-23T01:30:12.1518288Z Job defined at: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/master 2022-11-23T01:30:12.1518311Z Waiting for a runner to pick up this job... 2022-11-23T01:30:12.3452431Z Job is about to start running on the runner: worker-rocm-amd-90 (repository) 2022-11-23T01:30:15.9412274Z Current runner version: '2.299.1' 2022-11-23T01:30:15.9419185Z Runner name: 'worker-rocm-amd-90' 2022-11-23T01:30:15.9419785Z Runner group name: 'Default' 2022-11-23T01:30:15.9420833Z Machine name: 'jenkins-worker-rocm-amd-90' 2022-11-23T01:30:15.9423667Z ##[group]GITHUB_TOKEN Permissions 2022-11-23T01:30:15.9424493Z Actions: write 2022-11-23T01:30:15.9424873Z Checks: write 2022-11-23T01:30:15.9425246Z Contents: write 2022-11-23T01:30:15.9425644Z Deployments: write 2022-11-23T01:30:15.9426053Z Discussions: write 2022-11-23T01:30:15.9426461Z Issues: write 2022-11-23T01:30:15.9426840Z Metadata: read 2022-11-23T01:30:15.9427221Z Packages: write 2022-11-23T01:30:15.9427645Z Pages: write 2022-11-23T01:30:15.9428008Z PullRequests: write 2022-11-23T01:30:15.9428452Z RepositoryProjects: write 2022-11-23T01:30:15.9428893Z SecurityEvents: write 2022-11-23T01:30:15.9429313Z Statuses: write 2022-11-23T01:30:15.9429736Z ##[endgroup] 2022-11-23T01:30:15.9433194Z Secret source: Actions 2022-11-23T01:30:15.9433764Z Prepare workflow directory 2022-11-23T01:30:16.3392176Z Prepare all required actions 2022-11-23T01:30:16.3628549Z Getting action download info 2022-11-23T01:30:16.6588374Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f) 2022-11-23T01:30:26.5308263Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd) 2022-11-23T01:30:27.7551512Z Getting action download info 2022-11-23T01:30:28.0023941Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076) 2022-11-23T01:30:28.9916523Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml 2022-11-23T01:30:28.9918249Z ##[group] Inputs 2022-11-23T01:30:28.9918597Z build-environment: linux-focal-rocm5.2-py3.8 2022-11-23T01:30:28.9919123Z test-matrix: { include: [ { config: "distributed", shard: 1, num_shards: 2, runner: "linux.rocm.gpu" }, { config: "distributed", shard: 2, num_shards: 2, runner: "linux.rocm.gpu" }, ]} 2022-11-23T01:30:28.9919791Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:30:28.9920212Z sync-tag: 2022-11-23T01:30:28.9920442Z ##[endgroup] 2022-11-23T01:30:28.9921106Z Complete job name: linux-focal-rocm5.2-py3.8-distributed / test (distributed, 2, 2, linux.rocm.gpu, mem_leak_check) 2022-11-23T01:30:29.1123791Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master 2022-11-23T01:30:29.1124185Z with: 2022-11-23T01:30:29.1124409Z no-sudo: true 2022-11-23T01:30:29.1124643Z submodules: recursive 2022-11-23T01:30:29.1124887Z fetch-depth: 0 2022-11-23T01:30:29.1125112Z env: 2022-11-23T01:30:29.1125348Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:29.1125597Z ##[endgroup] 2022-11-23T01:30:29.1378296Z ##[group]Run retry () { 2022-11-23T01:30:29.1378623Z retry () { 2022-11-23T01:30:29.1378932Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2022-11-23T01:30:29.1379240Z } 2022-11-23T01:30:29.1379486Z echo "${GITHUB_WORKSPACE}" 2022-11-23T01:30:29.1379785Z if [ -z "${NO_SUDO}" ]; then 2022-11-23T01:30:29.1380106Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2022-11-23T01:30:29.1380403Z else 2022-11-23T01:30:29.1380730Z  retry rm -rf "${GITHUB_WORKSPACE}" 2022-11-23T01:30:29.1381021Z fi 2022-11-23T01:30:29.1381275Z mkdir "${GITHUB_WORKSPACE}" 2022-11-23T01:30:29.1406911Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:30:29.1407214Z env: 2022-11-23T01:30:29.1407456Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:29.1407713Z NO_SUDO: true 2022-11-23T01:30:29.1407944Z ##[endgroup] 2022-11-23T01:30:29.1687100Z /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:30.7193102Z ##[group]Run malfet/checkout@silent-checkout 2022-11-23T01:30:30.7193424Z with: 2022-11-23T01:30:30.7193684Z ref: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:30:30.7194134Z fetch-depth: 0 2022-11-23T01:30:30.7194371Z submodules: recursive 2022-11-23T01:30:30.7194622Z quiet-checkout: true 2022-11-23T01:30:30.7194896Z repository: pytorch/pytorch 2022-11-23T01:30:30.7195304Z token: *** 2022-11-23T01:30:30.7195551Z ssh-strict: true 2022-11-23T01:30:30.7195809Z persist-credentials: true 2022-11-23T01:30:30.7196056Z clean: true 2022-11-23T01:30:30.7196287Z lfs: false 2022-11-23T01:30:30.7196538Z set-safe-directory: true 2022-11-23T01:30:30.7196779Z env: 2022-11-23T01:30:30.7197021Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:30:30.7197262Z ##[endgroup] 2022-11-23T01:30:30.8723159Z Syncing repository: pytorch/pytorch 2022-11-23T01:30:30.8724982Z ##[group]Getting Git version info 2022-11-23T01:30:30.8725719Z Working directory is '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2022-11-23T01:30:30.8726445Z [command]/usr/bin/git version 2022-11-23T01:30:30.8726769Z git version 2.35.1 2022-11-23T01:30:30.8727618Z ##[endgroup] 2022-11-23T01:30:30.8742741Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/fffed1f8-ef37-47a4-8e9d-c5ede1a3398c' before making global git config changes 2022-11-23T01:30:30.8743307Z Adding repository directory to the temporary git global config as a safe directory 2022-11-23T01:30:30.8743962Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:30.8753832Z Deleting the contents of '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2022-11-23T01:30:30.8760507Z ##[group]Initializing the repository 2022-11-23T01:30:30.8766672Z [command]/usr/bin/git init /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:30:30.8814868Z hint: Using 'master' as the name for the initial branch. This default branch name 2022-11-23T01:30:30.8815504Z hint: is subject to change. To configure the initial branch name to use in all 2022-11-23T01:30:30.8816045Z hint: of your new repositories, which will suppress this warning, call: 2022-11-23T01:30:30.8819113Z hint: 2022-11-23T01:30:30.8820879Z hint: git config --global init.defaultBranch 2022-11-23T01:30:30.8821644Z hint: 2022-11-23T01:30:30.8822708Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2022-11-23T01:30:30.8824107Z hint: 'development'. The just-created branch can be renamed via this command: 2022-11-23T01:30:30.8825532Z hint: 2022-11-23T01:30:30.8826344Z hint: git branch -m 2022-11-23T01:30:30.8827678Z Initialized empty Git repository in /home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2022-11-23T01:30:30.8841642Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2022-11-23T01:30:30.8887994Z ##[endgroup] 2022-11-23T01:30:30.8889336Z ##[group]Disabling automatic garbage collection 2022-11-23T01:30:30.8894459Z [command]/usr/bin/git config --local gc.auto 0 2022-11-23T01:30:30.8951162Z ##[endgroup] 2022-11-23T01:30:30.8953341Z ##[group]Setting up auth 2022-11-23T01:30:30.8964278Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-11-23T01:30:30.9009482Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-11-23T01:30:30.9378361Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-11-23T01:30:30.9437639Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-11-23T01:30:30.9828162Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-11-23T01:30:30.9908866Z ##[endgroup] 2022-11-23T01:30:30.9910229Z ##[group]Fetching the repository 2022-11-23T01:30:30.9919929Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2022-11-23T01:31:33.7821431Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object} 2022-11-23T01:31:33.7879711Z 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:31:33.7893300Z ##[endgroup] 2022-11-23T01:31:33.7894973Z ##[group]Determining the checkout info 2022-11-23T01:31:33.7896443Z ##[endgroup] 2022-11-23T01:31:33.7897816Z ##[group]Checking out the ref 2022-11-23T01:31:33.7906782Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:31:35.1191268Z ##[endgroup] 2022-11-23T01:31:35.1192544Z ##[group]Setting up auth for fetching submodules 2022-11-23T01:31:35.1204958Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-11-23T01:31:35.1283491Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2022-11-23T01:31:35.1352688Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2022-11-23T01:31:35.1417898Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2022-11-23T01:31:35.1481511Z ##[endgroup] 2022-11-23T01:31:35.1483017Z ##[group]Fetching submodules 2022-11-23T01:31:35.1494279Z [command]/usr/bin/git submodule sync --recursive 2022-11-23T01:31:35.1903385Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2022-11-23T01:31:35.2284582Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2022-11-23T01:31:35.2286843Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2022-11-23T01:31:35.2289258Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2022-11-23T01:31:35.2293482Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2022-11-23T01:31:35.2297954Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2022-11-23T01:31:35.2303435Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2022-11-23T01:31:35.2308364Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2022-11-23T01:31:35.2313051Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2022-11-23T01:31:35.2318001Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2022-11-23T01:31:35.2323450Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2022-11-23T01:31:35.2328827Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2022-11-23T01:31:35.2334430Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2022-11-23T01:31:35.2340334Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2022-11-23T01:31:35.2345903Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2022-11-23T01:31:35.2351881Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2022-11-23T01:31:35.2357759Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2022-11-23T01:31:35.2363965Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2022-11-23T01:31:35.2370391Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:31:35.2377216Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2022-11-23T01:31:35.2383721Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2022-11-23T01:31:35.2387865Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2022-11-23T01:31:35.2392460Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2022-11-23T01:31:35.2396645Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2022-11-23T01:31:35.2401036Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2022-11-23T01:31:35.2405553Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2022-11-23T01:31:35.2410236Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2022-11-23T01:31:35.2415205Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2022-11-23T01:31:35.2419775Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2022-11-23T01:31:35.2424758Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2022-11-23T01:31:35.2429571Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2022-11-23T01:31:35.2434596Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2022-11-23T01:31:35.2439591Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2022-11-23T01:31:35.2444898Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2022-11-23T01:31:35.2449879Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2022-11-23T01:31:35.2456097Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2022-11-23T01:31:35.2461369Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2022-11-23T01:31:35.2466719Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2022-11-23T01:31:35.2471924Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2022-11-23T01:31:35.2477727Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2022-11-23T01:31:35.2483752Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2022-11-23T01:31:35.2489496Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2022-11-23T01:31:35.2595426Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2022-11-23T01:31:36.3589946Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2022-11-23T01:31:37.2843973Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2022-11-23T01:31:38.1209482Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2022-11-23T01:31:39.2716227Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2022-11-23T01:31:40.4738324Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2022-11-23T01:31:43.3363786Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2022-11-23T01:31:49.1834119Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2022-11-23T01:31:50.4793148Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2022-11-23T01:31:51.9024949Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2022-11-23T01:31:54.3343779Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2022-11-23T01:31:56.5461330Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2022-11-23T01:31:58.8837658Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2022-11-23T01:32:05.1162268Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2022-11-23T01:32:06.7217402Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2022-11-23T01:32:08.9530499Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2022-11-23T01:32:10.8777096Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2022-11-23T01:32:11.7476594Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2022-11-23T01:32:13.1113703Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2022-11-23T01:32:14.3222050Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2022-11-23T01:32:16.0878566Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2022-11-23T01:32:17.3611739Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2022-11-23T01:32:18.2212701Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2022-11-23T01:32:19.1855323Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2022-11-23T01:32:21.7713340Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2022-11-23T01:32:23.1823794Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2022-11-23T01:32:24.4719763Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2022-11-23T01:32:31.7112726Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2022-11-23T01:32:34.0421672Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2022-11-23T01:32:35.3766501Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2022-11-23T01:32:36.2643093Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2022-11-23T01:32:41.8091317Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2022-11-23T01:32:42.5062369Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2022-11-23T01:32:43.6162942Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2022-11-23T01:32:45.2948495Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2022-11-23T01:32:46.1700219Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2022-11-23T01:32:47.3651775Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2022-11-23T01:32:48.5620829Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2022-11-23T01:32:49.9986817Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2022-11-23T01:32:53.0693647Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2022-11-23T01:32:54.4648350Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2022-11-23T01:32:57.3495787Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2022-11-23T01:32:57.3924689Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2022-11-23T01:32:57.4288677Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2022-11-23T01:32:57.4851111Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2022-11-23T01:32:57.5358123Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2022-11-23T01:32:57.6052100Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2022-11-23T01:32:58.2156672Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2022-11-23T01:32:58.2688908Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-11-23T01:32:58.3977615Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c' 2022-11-23T01:32:58.4677116Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2022-11-23T01:32:58.7722889Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3' 2022-11-23T01:32:59.1914820Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a' 2022-11-23T01:32:59.4587090Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2022-11-23T01:32:59.5417366Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a' 2022-11-23T01:32:59.5487116Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:32:59.5489908Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:32:59.5494769Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:32:59.5499892Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:32:59.5570833Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2022-11-23T01:33:01.3228932Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2022-11-23T01:33:02.8820945Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2022-11-23T01:33:05.2700294Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2022-11-23T01:33:06.2984302Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2022-11-23T01:33:06.4287050Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2022-11-23T01:33:06.5337576Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2022-11-23T01:33:06.5721802Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5' 2022-11-23T01:33:06.6947613Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2022-11-23T01:33:06.7660483Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b' 2022-11-23T01:33:06.8056919Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2022-11-23T01:33:06.8769073Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2022-11-23T01:33:06.9326516Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195' 2022-11-23T01:33:07.0123285Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2022-11-23T01:33:07.0587353Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f' 2022-11-23T01:33:07.0660211Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:07.0731058Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2022-11-23T01:33:15.6151821Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d' 2022-11-23T01:33:15.6238044Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:15.6318578Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2022-11-23T01:33:24.4421948Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289' 2022-11-23T01:33:24.4839468Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2022-11-23T01:33:24.5281707Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2022-11-23T01:33:24.6498412Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562' 2022-11-23T01:33:24.6570110Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:24.6574794Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:24.6654615Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2022-11-23T01:33:26.5431362Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2022-11-23T01:33:28.3866221Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2022-11-23T01:33:28.4858366Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2022-11-23T01:33:28.5408570Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52' 2022-11-23T01:33:28.5886263Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2022-11-23T01:33:28.7204210Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2022-11-23T01:33:28.9950717Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b' 2022-11-23T01:33:29.0049282Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:29.0052966Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:29.0167278Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2022-11-23T01:33:31.1217748Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2022-11-23T01:33:32.8896312Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2022-11-23T01:33:32.9551310Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8' 2022-11-23T01:33:33.0041080Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2022-11-23T01:33:33.0118822Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:33.0186343Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2022-11-23T01:33:35.5733988Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2022-11-23T01:33:35.5812541Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:35.5815673Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:35.5911133Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2022-11-23T01:33:36.8046038Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2022-11-23T01:33:38.6059441Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-11-23T01:33:38.7214223Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2022-11-23T01:33:38.7293415Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:38.7369519Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2022-11-23T01:33:39.6669339Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-11-23T01:33:39.7070064Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2022-11-23T01:33:39.9922848Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2022-11-23T01:33:40.0002743Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:40.0006884Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:40.0091620Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2022-11-23T01:33:41.3282877Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2022-11-23T01:33:43.0282629Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2022-11-23T01:33:43.1396805Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2022-11-23T01:33:43.1775817Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2022-11-23T01:33:43.2201560Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2022-11-23T01:33:43.2914929Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917' 2022-11-23T01:33:43.3304986Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2022-11-23T01:33:43.3925849Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2022-11-23T01:33:43.4303659Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2022-11-23T01:33:43.5067933Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2022-11-23T01:33:43.6538414Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2022-11-23T01:33:43.7138974Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2022-11-23T01:33:43.7210255Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:43.7217926Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:43.7227120Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:43.7235907Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:43.7321026Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2022-11-23T01:33:45.5233573Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2022-11-23T01:33:46.6358347Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2022-11-23T01:33:48.8084141Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2022-11-23T01:33:50.8830134Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2022-11-23T01:33:50.9276284Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2022-11-23T01:33:51.0390836Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2022-11-23T01:33:51.1017967Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2022-11-23T01:33:51.1095039Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:51.1173456Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2022-11-23T01:33:51.9749186Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-11-23T01:33:52.1590898Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2022-11-23T01:33:52.1727962Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2022-11-23T01:33:52.2138212Z Entering 'android/libs/fbjni' 2022-11-23T01:33:52.2205574Z Entering 'third_party/FP16' 2022-11-23T01:33:52.2264256Z Entering 'third_party/FXdiv' 2022-11-23T01:33:52.2328806Z Entering 'third_party/NNPACK' 2022-11-23T01:33:52.2400813Z Entering 'third_party/QNNPACK' 2022-11-23T01:33:52.2463348Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:33:52.2533535Z Entering 'third_party/XNNPACK' 2022-11-23T01:33:52.2624224Z Entering 'third_party/benchmark' 2022-11-23T01:33:52.2683471Z Entering 'third_party/cpuinfo' 2022-11-23T01:33:52.2754965Z Entering 'third_party/cub' 2022-11-23T01:33:52.2829897Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:33:52.2905199Z Entering 'third_party/cutlass' 2022-11-23T01:33:52.2977031Z Entering 'third_party/eigen' 2022-11-23T01:33:52.3040850Z Entering 'third_party/fbgemm' 2022-11-23T01:33:52.3107679Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:52.3177488Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:52.3247429Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:52.3316953Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:52.3378680Z Entering 'third_party/flatbuffers' 2022-11-23T01:33:52.3445491Z Entering 'third_party/fmt' 2022-11-23T01:33:52.3517115Z Entering 'third_party/foxi' 2022-11-23T01:33:52.3589407Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:33:52.3646775Z Entering 'third_party/gloo' 2022-11-23T01:33:52.3712155Z Entering 'third_party/googletest' 2022-11-23T01:33:52.3784570Z Entering 'third_party/ideep' 2022-11-23T01:33:52.3847604Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:52.3919656Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:52.4006108Z Entering 'third_party/ios-cmake' 2022-11-23T01:33:52.4077394Z Entering 'third_party/ittapi' 2022-11-23T01:33:52.4144132Z Entering 'third_party/kineto' 2022-11-23T01:33:52.4215318Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:52.4287587Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:52.4356134Z Entering 'third_party/nccl/nccl' 2022-11-23T01:33:52.4421434Z Entering 'third_party/neon2sse' 2022-11-23T01:33:52.4492200Z Entering 'third_party/nlohmann' 2022-11-23T01:33:52.4562963Z Entering 'third_party/onnx' 2022-11-23T01:33:52.4654287Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:52.4723961Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:52.4789688Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:33:52.4853968Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:52.4929695Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:52.4997731Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:52.5067997Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:52.5139143Z Entering 'third_party/pocketfft' 2022-11-23T01:33:52.5210164Z Entering 'third_party/protobuf' 2022-11-23T01:33:52.5280954Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:52.5345985Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:52.5415161Z Entering 'third_party/psimd' 2022-11-23T01:33:52.5483951Z Entering 'third_party/pthreadpool' 2022-11-23T01:33:52.5543593Z Entering 'third_party/pybind11' 2022-11-23T01:33:52.5614253Z Entering 'third_party/python-enum' 2022-11-23T01:33:52.5680600Z Entering 'third_party/python-peachpy' 2022-11-23T01:33:52.5742181Z Entering 'third_party/python-six' 2022-11-23T01:33:52.5811765Z Entering 'third_party/sleef' 2022-11-23T01:33:52.5882996Z Entering 'third_party/tbb' 2022-11-23T01:33:52.5957244Z Entering 'third_party/tensorpipe' 2022-11-23T01:33:52.6037441Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:52.6103306Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:52.6173421Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:52.6243358Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:52.6312274Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:52.6376581Z Entering 'third_party/zstd' 2022-11-23T01:33:52.6454087Z ##[endgroup] 2022-11-23T01:33:52.6455462Z ##[group]Persisting credentials for submodules 2022-11-23T01:33:52.6469481Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2022-11-23T01:33:52.6879002Z Entering 'android/libs/fbjni' 2022-11-23T01:33:52.6946965Z Entering 'third_party/FP16' 2022-11-23T01:33:52.7001516Z Entering 'third_party/FXdiv' 2022-11-23T01:33:52.7057160Z Entering 'third_party/NNPACK' 2022-11-23T01:33:52.7122117Z Entering 'third_party/QNNPACK' 2022-11-23T01:33:52.7185601Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:33:52.7257189Z Entering 'third_party/XNNPACK' 2022-11-23T01:33:52.7348845Z Entering 'third_party/benchmark' 2022-11-23T01:33:52.7417154Z Entering 'third_party/cpuinfo' 2022-11-23T01:33:52.7486827Z Entering 'third_party/cub' 2022-11-23T01:33:52.7546809Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:33:52.7628470Z Entering 'third_party/cutlass' 2022-11-23T01:33:52.7712891Z Entering 'third_party/eigen' 2022-11-23T01:33:52.7778817Z Entering 'third_party/fbgemm' 2022-11-23T01:33:52.7847344Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:52.7913404Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:52.7977669Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:52.8035477Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:52.8107029Z Entering 'third_party/flatbuffers' 2022-11-23T01:33:52.8174217Z Entering 'third_party/fmt' 2022-11-23T01:33:52.8238794Z Entering 'third_party/foxi' 2022-11-23T01:33:52.8305564Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:33:52.8377733Z Entering 'third_party/gloo' 2022-11-23T01:33:52.8448570Z Entering 'third_party/googletest' 2022-11-23T01:33:52.8518553Z Entering 'third_party/ideep' 2022-11-23T01:33:52.8585099Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:52.8661633Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:52.8741973Z Entering 'third_party/ios-cmake' 2022-11-23T01:33:52.8809420Z Entering 'third_party/ittapi' 2022-11-23T01:33:52.8875189Z Entering 'third_party/kineto' 2022-11-23T01:33:52.8944137Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:52.9012567Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:52.9084336Z Entering 'third_party/nccl/nccl' 2022-11-23T01:33:52.9156330Z Entering 'third_party/neon2sse' 2022-11-23T01:33:52.9222566Z Entering 'third_party/nlohmann' 2022-11-23T01:33:52.9289907Z Entering 'third_party/onnx' 2022-11-23T01:33:52.9387520Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:52.9452506Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:52.9521416Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:33:52.9589960Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:52.9673848Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:52.9735773Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:52.9804190Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:52.9882215Z Entering 'third_party/pocketfft' 2022-11-23T01:33:52.9946717Z Entering 'third_party/protobuf' 2022-11-23T01:33:53.0021553Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:53.0092444Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:53.0158246Z Entering 'third_party/psimd' 2022-11-23T01:33:53.0228125Z Entering 'third_party/pthreadpool' 2022-11-23T01:33:53.0290806Z Entering 'third_party/pybind11' 2022-11-23T01:33:53.0359431Z Entering 'third_party/python-enum' 2022-11-23T01:33:53.0427178Z Entering 'third_party/python-peachpy' 2022-11-23T01:33:53.0494431Z Entering 'third_party/python-six' 2022-11-23T01:33:53.0558424Z Entering 'third_party/sleef' 2022-11-23T01:33:53.0623353Z Entering 'third_party/tbb' 2022-11-23T01:33:53.0694999Z Entering 'third_party/tensorpipe' 2022-11-23T01:33:53.0758341Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:53.0832727Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:53.0897002Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:53.0964613Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:53.1032494Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:53.1100222Z Entering 'third_party/zstd' 2022-11-23T01:33:53.1182583Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2022-11-23T01:33:53.1601145Z Entering 'android/libs/fbjni' 2022-11-23T01:33:53.1661084Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2022-11-23T01:33:53.1693344Z Entering 'third_party/FP16' 2022-11-23T01:33:53.1751430Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2022-11-23T01:33:53.1774820Z Entering 'third_party/FXdiv' 2022-11-23T01:33:53.1836927Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2022-11-23T01:33:53.1870779Z Entering 'third_party/NNPACK' 2022-11-23T01:33:53.1924198Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2022-11-23T01:33:53.1960715Z Entering 'third_party/QNNPACK' 2022-11-23T01:33:53.2016887Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2022-11-23T01:33:53.2052411Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:33:53.2115277Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2022-11-23T01:33:53.2147943Z Entering 'third_party/XNNPACK' 2022-11-23T01:33:53.2206212Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2022-11-23T01:33:53.2256741Z Entering 'third_party/benchmark' 2022-11-23T01:33:53.2315129Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:33:53.2347789Z Entering 'third_party/cpuinfo' 2022-11-23T01:33:53.2406697Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2022-11-23T01:33:53.2435310Z Entering 'third_party/cub' 2022-11-23T01:33:53.2487956Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2022-11-23T01:33:53.2519190Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:33:53.2574074Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2022-11-23T01:33:53.2617732Z Entering 'third_party/cutlass' 2022-11-23T01:33:53.2670589Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2022-11-23T01:33:53.2716421Z Entering 'third_party/eigen' 2022-11-23T01:33:53.2777819Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2022-11-23T01:33:53.2818683Z Entering 'third_party/fbgemm' 2022-11-23T01:33:53.2883098Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2022-11-23T01:33:53.2909342Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:53.2961379Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2022-11-23T01:33:53.2996206Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:53.3044225Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2022-11-23T01:33:53.3078137Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:53.3139067Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:33:53.3172879Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:53.3235567Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2022-11-23T01:33:53.3271346Z Entering 'third_party/flatbuffers' 2022-11-23T01:33:53.3322254Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2022-11-23T01:33:53.3363896Z Entering 'third_party/fmt' 2022-11-23T01:33:53.3425505Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2022-11-23T01:33:53.3459914Z Entering 'third_party/foxi' 2022-11-23T01:33:53.3514047Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2022-11-23T01:33:53.3535782Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:33:53.3592016Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2022-11-23T01:33:53.3625022Z Entering 'third_party/gloo' 2022-11-23T01:33:53.3683090Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2022-11-23T01:33:53.3717938Z Entering 'third_party/googletest' 2022-11-23T01:33:53.3783448Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:33:53.3817200Z Entering 'third_party/ideep' 2022-11-23T01:33:53.3874950Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2022-11-23T01:33:53.3908315Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:53.3959024Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2022-11-23T01:33:53.3996566Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:53.4055393Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2022-11-23T01:33:53.4094351Z Entering 'third_party/ios-cmake' 2022-11-23T01:33:53.4149310Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2022-11-23T01:33:53.4178415Z Entering 'third_party/ittapi' 2022-11-23T01:33:53.4238428Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2022-11-23T01:33:53.4273403Z Entering 'third_party/kineto' 2022-11-23T01:33:53.4335149Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2022-11-23T01:33:53.4367072Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:53.4427022Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2022-11-23T01:33:53.4459455Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:53.4522345Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2022-11-23T01:33:53.4560262Z Entering 'third_party/nccl/nccl' 2022-11-23T01:33:53.4616745Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2022-11-23T01:33:53.4646135Z Entering 'third_party/neon2sse' 2022-11-23T01:33:53.4704983Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2022-11-23T01:33:53.4737788Z Entering 'third_party/nlohmann' 2022-11-23T01:33:53.4793387Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2022-11-23T01:33:53.4832207Z Entering 'third_party/onnx' 2022-11-23T01:33:53.4893683Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2022-11-23T01:33:53.4940840Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:53.4999242Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:33:53.5032574Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:53.5096744Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:33:53.5133693Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:33:53.5197919Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2022-11-23T01:33:53.5219314Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:53.5281911Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2022-11-23T01:33:53.5314528Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:53.5369467Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:33:53.5403896Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:53.5472252Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:33:53.5505737Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:53.5571196Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-11-23T01:33:53.5619373Z Entering 'third_party/pocketfft' 2022-11-23T01:33:53.5675104Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2022-11-23T01:33:53.5712468Z Entering 'third_party/protobuf' 2022-11-23T01:33:53.5768211Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2022-11-23T01:33:53.5796435Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:53.5857909Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2022-11-23T01:33:53.5882589Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:53.5933856Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:33:53.5971711Z Entering 'third_party/psimd' 2022-11-23T01:33:53.6034527Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2022-11-23T01:33:53.6061015Z Entering 'third_party/pthreadpool' 2022-11-23T01:33:53.6122763Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2022-11-23T01:33:53.6155139Z Entering 'third_party/pybind11' 2022-11-23T01:33:53.6209734Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:33:53.6239594Z Entering 'third_party/python-enum' 2022-11-23T01:33:53.6300784Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2022-11-23T01:33:53.6334813Z Entering 'third_party/python-peachpy' 2022-11-23T01:33:53.6396419Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2022-11-23T01:33:53.6431211Z Entering 'third_party/python-six' 2022-11-23T01:33:53.6494929Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2022-11-23T01:33:53.6520786Z Entering 'third_party/sleef' 2022-11-23T01:33:53.6573773Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2022-11-23T01:33:53.6607038Z Entering 'third_party/tbb' 2022-11-23T01:33:53.6660936Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2022-11-23T01:33:53.6697968Z Entering 'third_party/tensorpipe' 2022-11-23T01:33:53.6760724Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2022-11-23T01:33:53.6794809Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:53.6845313Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2022-11-23T01:33:53.6878049Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:53.6937672Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2022-11-23T01:33:53.6971940Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:53.7035113Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2022-11-23T01:33:53.7068947Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:53.7126781Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2022-11-23T01:33:53.7158448Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:53.7220759Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-11-23T01:33:53.7252846Z Entering 'third_party/zstd' 2022-11-23T01:33:53.7305438Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2022-11-23T01:33:53.7666265Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2022-11-23T01:33:53.8079977Z Entering 'android/libs/fbjni' 2022-11-23T01:33:53.8146534Z Entering 'third_party/FP16' 2022-11-23T01:33:53.8212585Z Entering 'third_party/FXdiv' 2022-11-23T01:33:53.8284457Z Entering 'third_party/NNPACK' 2022-11-23T01:33:53.8349657Z Entering 'third_party/QNNPACK' 2022-11-23T01:33:53.8419473Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:33:53.8494195Z Entering 'third_party/XNNPACK' 2022-11-23T01:33:53.8577139Z Entering 'third_party/benchmark' 2022-11-23T01:33:53.8643621Z Entering 'third_party/cpuinfo' 2022-11-23T01:33:53.8710897Z Entering 'third_party/cub' 2022-11-23T01:33:53.8780322Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:33:53.8860042Z Entering 'third_party/cutlass' 2022-11-23T01:33:53.8933982Z Entering 'third_party/eigen' 2022-11-23T01:33:53.8996778Z Entering 'third_party/fbgemm' 2022-11-23T01:33:53.9056170Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:53.9128110Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:53.9197249Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:53.9271484Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:53.9336857Z Entering 'third_party/flatbuffers' 2022-11-23T01:33:53.9403001Z Entering 'third_party/fmt' 2022-11-23T01:33:53.9475308Z Entering 'third_party/foxi' 2022-11-23T01:33:53.9545074Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:33:53.9617073Z Entering 'third_party/gloo' 2022-11-23T01:33:53.9690099Z Entering 'third_party/googletest' 2022-11-23T01:33:53.9761497Z Entering 'third_party/ideep' 2022-11-23T01:33:53.9825574Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:53.9902571Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:53.9985678Z Entering 'third_party/ios-cmake' 2022-11-23T01:33:54.0052252Z Entering 'third_party/ittapi' 2022-11-23T01:33:54.0115302Z Entering 'third_party/kineto' 2022-11-23T01:33:54.0183180Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:54.0258684Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:54.0333553Z Entering 'third_party/nccl/nccl' 2022-11-23T01:33:54.0405178Z Entering 'third_party/neon2sse' 2022-11-23T01:33:54.0478599Z Entering 'third_party/nlohmann' 2022-11-23T01:33:54.0547734Z Entering 'third_party/onnx' 2022-11-23T01:33:54.0647499Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:54.0719634Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:54.0794882Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:33:54.0862399Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:54.0940409Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:54.1011812Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:54.1082918Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:54.1171239Z Entering 'third_party/pocketfft' 2022-11-23T01:33:54.1239093Z Entering 'third_party/protobuf' 2022-11-23T01:33:54.1313674Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:54.1375765Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:54.1445319Z Entering 'third_party/psimd' 2022-11-23T01:33:54.1512522Z Entering 'third_party/pthreadpool' 2022-11-23T01:33:54.1587603Z Entering 'third_party/pybind11' 2022-11-23T01:33:54.1656480Z Entering 'third_party/python-enum' 2022-11-23T01:33:54.1721943Z Entering 'third_party/python-peachpy' 2022-11-23T01:33:54.1776700Z Entering 'third_party/python-six' 2022-11-23T01:33:54.1847599Z Entering 'third_party/sleef' 2022-11-23T01:33:54.1920665Z Entering 'third_party/tbb' 2022-11-23T01:33:54.1996133Z Entering 'third_party/tensorpipe' 2022-11-23T01:33:54.2065152Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:54.2140023Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:54.2214147Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:54.2284170Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:54.2351004Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:54.2417555Z Entering 'third_party/zstd' 2022-11-23T01:33:54.2518162Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2022-11-23T01:33:54.2922072Z Entering 'android/libs/fbjni' 2022-11-23T01:33:54.2989915Z Entering 'third_party/FP16' 2022-11-23T01:33:54.3062822Z Entering 'third_party/FXdiv' 2022-11-23T01:33:54.3129352Z Entering 'third_party/NNPACK' 2022-11-23T01:33:54.3203125Z Entering 'third_party/QNNPACK' 2022-11-23T01:33:54.3262840Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T01:33:54.3327188Z Entering 'third_party/XNNPACK' 2022-11-23T01:33:54.3417722Z Entering 'third_party/benchmark' 2022-11-23T01:33:54.3484547Z Entering 'third_party/cpuinfo' 2022-11-23T01:33:54.3542638Z Entering 'third_party/cub' 2022-11-23T01:33:54.3615536Z Entering 'third_party/cudnn_frontend' 2022-11-23T01:33:54.3697191Z Entering 'third_party/cutlass' 2022-11-23T01:33:54.3778867Z Entering 'third_party/eigen' 2022-11-23T01:33:54.3848141Z Entering 'third_party/fbgemm' 2022-11-23T01:33:54.3915156Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T01:33:54.3983688Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T01:33:54.4040540Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T01:33:54.4111412Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T01:33:54.4183407Z Entering 'third_party/flatbuffers' 2022-11-23T01:33:54.4244165Z Entering 'third_party/fmt' 2022-11-23T01:33:54.4302164Z Entering 'third_party/foxi' 2022-11-23T01:33:54.4368212Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T01:33:54.4438698Z Entering 'third_party/gloo' 2022-11-23T01:33:54.4513778Z Entering 'third_party/googletest' 2022-11-23T01:33:54.4577945Z Entering 'third_party/ideep' 2022-11-23T01:33:54.4648391Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T01:33:54.4718292Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T01:33:54.4806488Z Entering 'third_party/ios-cmake' 2022-11-23T01:33:54.4867370Z Entering 'third_party/ittapi' 2022-11-23T01:33:54.4941089Z Entering 'third_party/kineto' 2022-11-23T01:33:54.5012811Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T01:33:54.5079451Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T01:33:54.5155832Z Entering 'third_party/nccl/nccl' 2022-11-23T01:33:54.5222891Z Entering 'third_party/neon2sse' 2022-11-23T01:33:54.5293684Z Entering 'third_party/nlohmann' 2022-11-23T01:33:54.5364787Z Entering 'third_party/onnx' 2022-11-23T01:33:54.5461418Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T01:33:54.5534356Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T01:33:54.5610363Z Entering 'third_party/onnx-tensorrt' 2022-11-23T01:33:54.5680648Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T01:33:54.5764358Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T01:33:54.5831941Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T01:33:54.5903334Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T01:33:54.5988516Z Entering 'third_party/pocketfft' 2022-11-23T01:33:54.6053268Z Entering 'third_party/protobuf' 2022-11-23T01:33:54.6128490Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T01:33:54.6189601Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T01:33:54.6254625Z Entering 'third_party/psimd' 2022-11-23T01:33:54.6321837Z Entering 'third_party/pthreadpool' 2022-11-23T01:33:54.6386601Z Entering 'third_party/pybind11' 2022-11-23T01:33:54.6452636Z Entering 'third_party/python-enum' 2022-11-23T01:33:54.6524152Z Entering 'third_party/python-peachpy' 2022-11-23T01:33:54.6599009Z Entering 'third_party/python-six' 2022-11-23T01:33:54.6672359Z Entering 'third_party/sleef' 2022-11-23T01:33:54.6729259Z Entering 'third_party/tbb' 2022-11-23T01:33:54.6788688Z Entering 'third_party/tensorpipe' 2022-11-23T01:33:54.6863921Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T01:33:54.6926106Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T01:33:54.6997013Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T01:33:54.7070619Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T01:33:54.7140506Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T01:33:54.7213770Z Entering 'third_party/zstd' 2022-11-23T01:33:54.7304121Z ##[endgroup] 2022-11-23T01:33:54.7383554Z [command]/usr/bin/git log -1 --format='%H' 2022-11-23T01:33:54.7443715Z '1cfd3858ac54fe3883534309081631a0a892ba3f' 2022-11-23T01:33:54.7790224Z Prepare all required actions 2022-11-23T01:33:54.7841066Z ##[group]Run ./.github/actions/setup-rocm 2022-11-23T01:33:54.7841335Z env: 2022-11-23T01:33:54.7841570Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:54.7841823Z ##[endgroup] 2022-11-23T01:33:54.7861101Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2022-11-23T01:33:54.7861621Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2022-11-23T01:33:54.7882761Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:54.7883043Z env: 2022-11-23T01:33:54.7883276Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:54.7883521Z ##[endgroup] 2022-11-23T01:33:54.7993360Z ##[group]Run cat /etc/os-release || true 2022-11-23T01:33:54.7994046Z cat /etc/os-release || true 2022-11-23T01:33:54.7994756Z cat /etc/apt/sources.list.d/rocm.list || true 2022-11-23T01:33:54.7995482Z cat /opt/rocm/.info/version || true 2022-11-23T01:33:54.7996120Z whoami 2022-11-23T01:33:54.8036916Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:54.8037536Z env: 2022-11-23T01:33:54.8038057Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:54.8038730Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:54.8039333Z ##[endgroup] 2022-11-23T01:33:54.8101635Z NAME="Ubuntu" 2022-11-23T01:33:54.8102393Z VERSION="18.04.5 LTS (Bionic Beaver)" 2022-11-23T01:33:54.8107557Z ID=ubuntu 2022-11-23T01:33:54.8108155Z ID_LIKE=debian 2022-11-23T01:33:54.8109214Z PRETTY_NAME="Ubuntu 18.04.5 LTS" 2022-11-23T01:33:54.8110151Z VERSION_ID="18.04" 2022-11-23T01:33:54.8111154Z HOME_URL="https://www.ubuntu.com/" 2022-11-23T01:33:54.8114704Z SUPPORT_URL="https://help.ubuntu.com/" 2022-11-23T01:33:54.8115982Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-11-23T01:33:54.8117957Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-11-23T01:33:54.8119058Z VERSION_CODENAME=bionic 2022-11-23T01:33:54.8119841Z UBUNTU_CODENAME=bionic 2022-11-23T01:33:54.8120807Z deb [arch=amd64] http://repo.radeon.com/rocm/apt/4.2 xenial main 2022-11-23T01:33:54.8126693Z 4.2.0-21 2022-11-23T01:33:54.8143915Z pytorchci 2022-11-23T01:33:54.8189817Z ##[group]Run rocm-smi 2022-11-23T01:33:54.8190520Z rocm-smi 2022-11-23T01:33:54.8236952Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:54.8237489Z env: 2022-11-23T01:33:54.8237919Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:54.8238504Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:54.8238996Z ##[endgroup] 2022-11-23T01:33:54.9187505Z 2022-11-23T01:33:54.9187807Z 2022-11-23T01:33:54.9188536Z ======================= ROCm System Management Interface ======================= 2022-11-23T01:33:54.9189880Z ================================= Concise Info ================================= 2022-11-23T01:33:54.9191132Z GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2022-11-23T01:33:54.9192224Z 0 34.0c 18.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:33:54.9193058Z 1 32.0c 18.0W 930Mhz 350Mhz 255% auto 225.0W 0% 0% 2022-11-23T01:33:54.9193851Z ================================================================================ 2022-11-23T01:33:54.9194660Z ============================= End of ROCm SMI Log ============================== 2022-11-23T01:33:54.9298144Z ##[group]Run rocminfo 2022-11-23T01:33:54.9298804Z rocminfo 2022-11-23T01:33:54.9348735Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:54.9349467Z env: 2022-11-23T01:33:54.9350079Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:54.9350875Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:54.9351598Z ##[endgroup] 2022-11-23T01:33:55.0249606Z ROCk module is loaded 2022-11-23T01:33:55.0250755Z ===================== 2022-11-23T01:33:55.0251713Z HSA System Attributes 2022-11-23T01:33:55.0252632Z ===================== 2022-11-23T01:33:55.0254254Z Runtime Version: 1.1 2022-11-23T01:33:55.0255370Z System Timestamp Freq.: 1000.000000MHz 2022-11-23T01:33:55.0256637Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2022-11-23T01:33:55.0257864Z Machine Model: LARGE 2022-11-23T01:33:55.0258958Z System Endianness: LITTLE 2022-11-23T01:33:55.0259827Z 2022-11-23T01:33:55.0260170Z ========== 2022-11-23T01:33:55.0261141Z HSA Agents 2022-11-23T01:33:55.0262021Z ========== 2022-11-23T01:33:55.0262859Z ******* 2022-11-23T01:33:55.0263702Z Agent 1 2022-11-23T01:33:55.0264549Z ******* 2022-11-23T01:33:55.0266078Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:55.0267491Z Uuid: CPU-XX 2022-11-23T01:33:55.0268941Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:55.0270091Z Vendor Name: CPU 2022-11-23T01:33:55.0270920Z Feature: None specified 2022-11-23T01:33:55.0271762Z Profile: FULL_PROFILE 2022-11-23T01:33:55.0272571Z Float Round Mode: NEAR 2022-11-23T01:33:55.0273371Z Max Queue Number: 0(0x0) 2022-11-23T01:33:55.0274165Z Queue Min Size: 0(0x0) 2022-11-23T01:33:55.0275253Z Queue Max Size: 0(0x0) 2022-11-23T01:33:55.0276059Z Queue Type: MULTI 2022-11-23T01:33:55.0276821Z Node: 0 2022-11-23T01:33:55.0277562Z Device Type: CPU 2022-11-23T01:33:55.0278259Z Cache Info: 2022-11-23T01:33:55.0278984Z L1: 32768(0x8000) KB 2022-11-23T01:33:55.0279740Z Chip ID: 0(0x0) 2022-11-23T01:33:55.0280515Z Cacheline Size: 64(0x40) 2022-11-23T01:33:55.0281309Z Max Clock Freq. (MHz): 2200 2022-11-23T01:33:55.0282069Z BDFID: 0 2022-11-23T01:33:55.0282813Z Internal Node ID: 0 2022-11-23T01:33:55.0283591Z Compute Unit: 16 2022-11-23T01:33:55.0284367Z SIMDs per CU: 0 2022-11-23T01:33:55.0285147Z Shader Engines: 0 2022-11-23T01:33:55.0285945Z Shader Arrs. per Eng.: 0 2022-11-23T01:33:55.0286765Z WatchPts on Addr. Ranges:1 2022-11-23T01:33:55.0287482Z Features: None 2022-11-23T01:33:55.0288129Z Pool Info: 2022-11-23T01:33:55.0288765Z Pool 1 2022-11-23T01:33:55.0289555Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:33:55.0290412Z Size: 65894484(0x3ed7854) KB 2022-11-23T01:33:55.0291408Z Allocatable: TRUE 2022-11-23T01:33:55.0292208Z Alloc Granule: 4KB 2022-11-23T01:33:55.0293028Z Alloc Alignment: 4KB 2022-11-23T01:33:55.0293862Z Accessible by all: TRUE 2022-11-23T01:33:55.0294579Z Pool 2 2022-11-23T01:33:55.0295366Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:55.0296202Z Size: 65894484(0x3ed7854) KB 2022-11-23T01:33:55.0297153Z Allocatable: TRUE 2022-11-23T01:33:55.0297969Z Alloc Granule: 4KB 2022-11-23T01:33:55.0298793Z Alloc Alignment: 4KB 2022-11-23T01:33:55.0299614Z Accessible by all: TRUE 2022-11-23T01:33:55.0300321Z ISA Info: 2022-11-23T01:33:55.0300941Z N/A 2022-11-23T01:33:55.0301517Z ******* 2022-11-23T01:33:55.0302115Z Agent 2 2022-11-23T01:33:55.0302701Z ******* 2022-11-23T01:33:55.0303728Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:55.0304768Z Uuid: CPU-XX 2022-11-23T01:33:55.0305837Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:55.0306672Z Vendor Name: CPU 2022-11-23T01:33:55.0307484Z Feature: None specified 2022-11-23T01:33:55.0308313Z Profile: FULL_PROFILE 2022-11-23T01:33:55.0309122Z Float Round Mode: NEAR 2022-11-23T01:33:55.0309922Z Max Queue Number: 0(0x0) 2022-11-23T01:33:55.0310704Z Queue Min Size: 0(0x0) 2022-11-23T01:33:55.0311469Z Queue Max Size: 0(0x0) 2022-11-23T01:33:55.0312398Z Queue Type: MULTI 2022-11-23T01:33:55.0313147Z Node: 1 2022-11-23T01:33:55.0313902Z Device Type: CPU 2022-11-23T01:33:55.0314589Z Cache Info: 2022-11-23T01:33:55.0315302Z L1: 32768(0x8000) KB 2022-11-23T01:33:55.0316056Z Chip ID: 0(0x0) 2022-11-23T01:33:55.0316821Z Cacheline Size: 64(0x40) 2022-11-23T01:33:55.0317607Z Max Clock Freq. (MHz): 2200 2022-11-23T01:33:55.0318369Z BDFID: 0 2022-11-23T01:33:55.0319127Z Internal Node ID: 1 2022-11-23T01:33:55.0319902Z Compute Unit: 16 2022-11-23T01:33:55.0320670Z SIMDs per CU: 0 2022-11-23T01:33:55.0321440Z Shader Engines: 0 2022-11-23T01:33:55.0322235Z Shader Arrs. per Eng.: 0 2022-11-23T01:33:55.0323053Z WatchPts on Addr. Ranges:1 2022-11-23T01:33:55.0323779Z Features: None 2022-11-23T01:33:55.0324422Z Pool Info: 2022-11-23T01:33:55.0325058Z Pool 1 2022-11-23T01:33:55.0325838Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:33:55.0326688Z Size: 66027676(0x3ef809c) KB 2022-11-23T01:33:55.0327487Z Allocatable: TRUE 2022-11-23T01:33:55.0328294Z Alloc Granule: 4KB 2022-11-23T01:33:55.0329100Z Alloc Alignment: 4KB 2022-11-23T01:33:55.0329917Z Accessible by all: TRUE 2022-11-23T01:33:55.0330991Z Pool 2 2022-11-23T01:33:55.0331786Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:55.0332625Z Size: 66027676(0x3ef809c) KB 2022-11-23T01:33:55.0333419Z Allocatable: TRUE 2022-11-23T01:33:55.0334227Z Alloc Granule: 4KB 2022-11-23T01:33:55.0335458Z Alloc Alignment: 4KB 2022-11-23T01:33:55.0336293Z Accessible by all: TRUE 2022-11-23T01:33:55.0337009Z ISA Info: 2022-11-23T01:33:55.0337668Z N/A 2022-11-23T01:33:55.0338289Z ******* 2022-11-23T01:33:55.0338869Z Agent 3 2022-11-23T01:33:55.0339473Z ******* 2022-11-23T01:33:55.0340507Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:55.0341578Z Uuid: CPU-XX 2022-11-23T01:33:55.0342675Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:55.0343530Z Vendor Name: CPU 2022-11-23T01:33:55.0344360Z Feature: None specified 2022-11-23T01:33:55.0345201Z Profile: FULL_PROFILE 2022-11-23T01:33:55.0346044Z Float Round Mode: NEAR 2022-11-23T01:33:55.0346853Z Max Queue Number: 0(0x0) 2022-11-23T01:33:55.0347275Z Queue Min Size: 0(0x0) 2022-11-23T01:33:55.0347650Z Queue Max Size: 0(0x0) 2022-11-23T01:33:55.0347951Z Queue Type: MULTI 2022-11-23T01:33:55.0348250Z Node: 2 2022-11-23T01:33:55.0348605Z Device Type: CPU 2022-11-23T01:33:55.0348876Z Cache Info: 2022-11-23T01:33:55.0349152Z L1: 32768(0x8000) KB 2022-11-23T01:33:55.0349442Z Chip ID: 0(0x0) 2022-11-23T01:33:55.0349730Z Cacheline Size: 64(0x40) 2022-11-23T01:33:55.0350040Z Max Clock Freq. (MHz): 2200 2022-11-23T01:33:55.0350333Z BDFID: 0 2022-11-23T01:33:55.0350636Z Internal Node ID: 2 2022-11-23T01:33:55.0350936Z Compute Unit: 16 2022-11-23T01:33:55.0351237Z SIMDs per CU: 0 2022-11-23T01:33:55.0351550Z Shader Engines: 0 2022-11-23T01:33:55.0351860Z Shader Arrs. per Eng.: 0 2022-11-23T01:33:55.0352183Z WatchPts on Addr. Ranges:1 2022-11-23T01:33:55.0352464Z Features: None 2022-11-23T01:33:55.0352706Z Pool Info: 2022-11-23T01:33:55.0352961Z Pool 1 2022-11-23T01:33:55.0353265Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:33:55.0353600Z Size: 66052532(0x3efe1b4) KB 2022-11-23T01:33:55.0353909Z Allocatable: TRUE 2022-11-23T01:33:55.0354222Z Alloc Granule: 4KB 2022-11-23T01:33:55.0354523Z Alloc Alignment: 4KB 2022-11-23T01:33:55.0354842Z Accessible by all: TRUE 2022-11-23T01:33:55.0355121Z Pool 2 2022-11-23T01:33:55.0355424Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:55.0355752Z Size: 66052532(0x3efe1b4) KB 2022-11-23T01:33:55.0356061Z Allocatable: TRUE 2022-11-23T01:33:55.0356361Z Alloc Granule: 4KB 2022-11-23T01:33:55.0356672Z Alloc Alignment: 4KB 2022-11-23T01:33:55.0357036Z Accessible by all: TRUE 2022-11-23T01:33:55.0357318Z ISA Info: 2022-11-23T01:33:55.0357562Z N/A 2022-11-23T01:33:55.0357796Z ******* 2022-11-23T01:33:55.0358010Z Agent 4 2022-11-23T01:33:55.0358232Z ******* 2022-11-23T01:33:55.0358611Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:55.0359004Z Uuid: CPU-XX 2022-11-23T01:33:55.0359409Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:33:55.0359732Z Vendor Name: CPU 2022-11-23T01:33:55.0360031Z Feature: None specified 2022-11-23T01:33:55.0360346Z Profile: FULL_PROFILE 2022-11-23T01:33:55.0360657Z Float Round Mode: NEAR 2022-11-23T01:33:55.0360957Z Max Queue Number: 0(0x0) 2022-11-23T01:33:55.0361256Z Queue Min Size: 0(0x0) 2022-11-23T01:33:55.0361552Z Queue Max Size: 0(0x0) 2022-11-23T01:33:55.0361839Z Queue Type: MULTI 2022-11-23T01:33:55.0362129Z Node: 3 2022-11-23T01:33:55.0362415Z Device Type: CPU 2022-11-23T01:33:55.0362677Z Cache Info: 2022-11-23T01:33:55.0363002Z L1: 32768(0x8000) KB 2022-11-23T01:33:55.0363293Z Chip ID: 0(0x0) 2022-11-23T01:33:55.0363578Z Cacheline Size: 64(0x40) 2022-11-23T01:33:55.0363885Z Max Clock Freq. (MHz): 2200 2022-11-23T01:33:55.0364173Z BDFID: 0 2022-11-23T01:33:55.0364466Z Internal Node ID: 3 2022-11-23T01:33:55.0364760Z Compute Unit: 16 2022-11-23T01:33:55.0365052Z SIMDs per CU: 0 2022-11-23T01:33:55.0365348Z Shader Engines: 0 2022-11-23T01:33:55.0365641Z Shader Arrs. per Eng.: 0 2022-11-23T01:33:55.0365950Z WatchPts on Addr. Ranges:1 2022-11-23T01:33:55.0366227Z Features: None 2022-11-23T01:33:55.0366470Z Pool Info: 2022-11-23T01:33:55.0366713Z Pool 1 2022-11-23T01:33:55.0367013Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:33:55.0367327Z Size: 66051008(0x3efdbc0) KB 2022-11-23T01:33:55.0367632Z Allocatable: TRUE 2022-11-23T01:33:55.0367939Z Alloc Granule: 4KB 2022-11-23T01:33:55.0368245Z Alloc Alignment: 4KB 2022-11-23T01:33:55.0368553Z Accessible by all: TRUE 2022-11-23T01:33:55.0368820Z Pool 2 2022-11-23T01:33:55.0369108Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:55.0369429Z Size: 66051008(0x3efdbc0) KB 2022-11-23T01:33:55.0369733Z Allocatable: TRUE 2022-11-23T01:33:55.0370040Z Alloc Granule: 4KB 2022-11-23T01:33:55.0370345Z Alloc Alignment: 4KB 2022-11-23T01:33:55.0370720Z Accessible by all: TRUE 2022-11-23T01:33:55.0370982Z ISA Info: 2022-11-23T01:33:55.0371217Z N/A 2022-11-23T01:33:55.0371550Z ******* 2022-11-23T01:33:55.0371818Z Agent 5 2022-11-23T01:33:55.0372090Z ******* 2022-11-23T01:33:55.0372396Z Name: gfx906 2022-11-23T01:33:55.0372863Z Uuid: GPU-974210a172e62127 2022-11-23T01:33:55.0373223Z Marketing Name: Vega 20 2022-11-23T01:33:55.0373580Z Vendor Name: AMD 2022-11-23T01:33:55.0373952Z Feature: KERNEL_DISPATCH 2022-11-23T01:33:55.0374324Z Profile: BASE_PROFILE 2022-11-23T01:33:55.0374699Z Float Round Mode: NEAR 2022-11-23T01:33:55.0375055Z Max Queue Number: 128(0x80) 2022-11-23T01:33:55.0375409Z Queue Min Size: 4096(0x1000) 2022-11-23T01:33:55.0375775Z Queue Max Size: 131072(0x20000) 2022-11-23T01:33:55.0376131Z Queue Type: MULTI 2022-11-23T01:33:55.0376475Z Node: 4 2022-11-23T01:33:55.0376823Z Device Type: GPU 2022-11-23T01:33:55.0377130Z Cache Info: 2022-11-23T01:33:55.0377451Z L1: 16(0x10) KB 2022-11-23T01:33:55.0377791Z Chip ID: 26273(0x66a1) 2022-11-23T01:33:55.0378259Z Cacheline Size: 64(0x40) 2022-11-23T01:33:55.0378620Z Max Clock Freq. (MHz): 1725 2022-11-23T01:33:55.0378963Z BDFID: 8960 2022-11-23T01:33:55.0379310Z Internal Node ID: 4 2022-11-23T01:33:55.0379656Z Compute Unit: 60 2022-11-23T01:33:55.0380008Z SIMDs per CU: 4 2022-11-23T01:33:55.0380361Z Shader Engines: 4 2022-11-23T01:33:55.0380720Z Shader Arrs. per Eng.: 1 2022-11-23T01:33:55.0381099Z WatchPts on Addr. Ranges:4 2022-11-23T01:33:55.0381453Z Features: KERNEL_DISPATCH 2022-11-23T01:33:55.0381811Z Fast F16 Operation: FALSE 2022-11-23T01:33:55.0382182Z Wavefront Size: 64(0x40) 2022-11-23T01:33:55.0382547Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:55.0382897Z Workgroup Max Size per Dimension: 2022-11-23T01:33:55.0383243Z x 1024(0x400) 2022-11-23T01:33:55.0383580Z y 1024(0x400) 2022-11-23T01:33:55.0383907Z z 1024(0x400) 2022-11-23T01:33:55.0384302Z Max Waves Per CU: 40(0x28) 2022-11-23T01:33:55.0384862Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:33:55.0385243Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:55.0385703Z Grid Max Size per Dimension: 2022-11-23T01:33:55.0386054Z x 4294967295(0xffffffff) 2022-11-23T01:33:55.0386405Z y 4294967295(0xffffffff) 2022-11-23T01:33:55.0386762Z z 4294967295(0xffffffff) 2022-11-23T01:33:55.0387118Z Max fbarriers/Workgrp: 32 2022-11-23T01:33:55.0387442Z Pool Info: 2022-11-23T01:33:55.0387731Z Pool 1 2022-11-23T01:33:55.0388086Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:55.0388567Z Size: 33538048(0x1ffc000) KB 2022-11-23T01:33:55.0388933Z Allocatable: TRUE 2022-11-23T01:33:55.0389295Z Alloc Granule: 4KB 2022-11-23T01:33:55.0389661Z Alloc Alignment: 4KB 2022-11-23T01:33:55.0390042Z Accessible by all: FALSE 2022-11-23T01:33:55.0390367Z Pool 2 2022-11-23T01:33:55.0390703Z Segment: GROUP 2022-11-23T01:33:55.0391059Z Size: 64(0x40) KB 2022-11-23T01:33:55.0391419Z Allocatable: FALSE 2022-11-23T01:33:55.0391785Z Alloc Granule: 0KB 2022-11-23T01:33:55.0392151Z Alloc Alignment: 0KB 2022-11-23T01:33:55.0392529Z Accessible by all: FALSE 2022-11-23T01:33:55.0392856Z ISA Info: 2022-11-23T01:33:55.0393137Z ISA 1 2022-11-23T01:33:55.0393641Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:33:55.0394083Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:33:55.0394490Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:33:55.0394885Z Default Rounding Mode: NEAR 2022-11-23T01:33:55.0395339Z Default Rounding Mode: NEAR 2022-11-23T01:33:55.0395711Z Fast f16: TRUE 2022-11-23T01:33:55.0396071Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:55.0396417Z Workgroup Max Size per Dimension: 2022-11-23T01:33:55.0396772Z x 1024(0x400) 2022-11-23T01:33:55.0397103Z y 1024(0x400) 2022-11-23T01:33:55.0397385Z z 1024(0x400) 2022-11-23T01:33:55.0397689Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:55.0397982Z Grid Max Size per Dimension: 2022-11-23T01:33:55.0398267Z x 4294967295(0xffffffff) 2022-11-23T01:33:55.0398567Z y 4294967295(0xffffffff) 2022-11-23T01:33:55.0398871Z z 4294967295(0xffffffff) 2022-11-23T01:33:55.0399178Z FBarrier Max Size: 32 2022-11-23T01:33:55.0399433Z ******* 2022-11-23T01:33:55.0399657Z Agent 6 2022-11-23T01:33:55.0399872Z ******* 2022-11-23T01:33:55.0400132Z Name: gfx906 2022-11-23T01:33:55.0400527Z Uuid: GPU-eb9c20c172e620f6 2022-11-23T01:33:55.0400835Z Marketing Name: Vega 20 2022-11-23T01:33:55.0401135Z Vendor Name: AMD 2022-11-23T01:33:55.0401443Z Feature: KERNEL_DISPATCH 2022-11-23T01:33:55.0401751Z Profile: BASE_PROFILE 2022-11-23T01:33:55.0402061Z Float Round Mode: NEAR 2022-11-23T01:33:55.0402364Z Max Queue Number: 128(0x80) 2022-11-23T01:33:55.0402669Z Queue Min Size: 4096(0x1000) 2022-11-23T01:33:55.0402969Z Queue Max Size: 131072(0x20000) 2022-11-23T01:33:55.0403271Z Queue Type: MULTI 2022-11-23T01:33:55.0403559Z Node: 5 2022-11-23T01:33:55.0403892Z Device Type: GPU 2022-11-23T01:33:55.0404156Z Cache Info: 2022-11-23T01:33:55.0404426Z L1: 16(0x10) KB 2022-11-23T01:33:55.0404713Z Chip ID: 26273(0x66a1) 2022-11-23T01:33:55.0405007Z Cacheline Size: 64(0x40) 2022-11-23T01:33:55.0405308Z Max Clock Freq. (MHz): 1725 2022-11-23T01:33:55.0405591Z BDFID: 25344 2022-11-23T01:33:55.0405883Z Internal Node ID: 5 2022-11-23T01:33:55.0406178Z Compute Unit: 60 2022-11-23T01:33:55.0406469Z SIMDs per CU: 4 2022-11-23T01:33:55.0406764Z Shader Engines: 4 2022-11-23T01:33:55.0407066Z Shader Arrs. per Eng.: 1 2022-11-23T01:33:55.0407376Z WatchPts on Addr. Ranges:4 2022-11-23T01:33:55.0407673Z Features: KERNEL_DISPATCH 2022-11-23T01:33:55.0407980Z Fast F16 Operation: FALSE 2022-11-23T01:33:55.0408287Z Wavefront Size: 64(0x40) 2022-11-23T01:33:55.0408593Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:55.0408884Z Workgroup Max Size per Dimension: 2022-11-23T01:33:55.0409167Z x 1024(0x400) 2022-11-23T01:33:55.0409491Z y 1024(0x400) 2022-11-23T01:33:55.0409766Z z 1024(0x400) 2022-11-23T01:33:55.0410060Z Max Waves Per CU: 40(0x28) 2022-11-23T01:33:55.0410470Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:33:55.0410907Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:55.0411225Z Grid Max Size per Dimension: 2022-11-23T01:33:55.0411602Z x 4294967295(0xffffffff) 2022-11-23T01:33:55.0411978Z y 4294967295(0xffffffff) 2022-11-23T01:33:55.0412364Z z 4294967295(0xffffffff) 2022-11-23T01:33:55.0412755Z Max fbarriers/Workgrp: 32 2022-11-23T01:33:55.0413109Z Pool Info: 2022-11-23T01:33:55.0413429Z Pool 1 2022-11-23T01:33:55.0413804Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:33:55.0414222Z Size: 33538048(0x1ffc000) KB 2022-11-23T01:33:55.0414621Z Allocatable: TRUE 2022-11-23T01:33:55.0415023Z Alloc Granule: 4KB 2022-11-23T01:33:55.0415427Z Alloc Alignment: 4KB 2022-11-23T01:33:55.0415844Z Accessible by all: FALSE 2022-11-23T01:33:55.0416190Z Pool 2 2022-11-23T01:33:55.0416557Z Segment: GROUP 2022-11-23T01:33:55.0416940Z Size: 64(0x40) KB 2022-11-23T01:33:55.0417339Z Allocatable: FALSE 2022-11-23T01:33:55.0417817Z Alloc Granule: 0KB 2022-11-23T01:33:55.0418203Z Alloc Alignment: 0KB 2022-11-23T01:33:55.0418618Z Accessible by all: FALSE 2022-11-23T01:33:55.0418955Z ISA Info: 2022-11-23T01:33:55.0419265Z ISA 1 2022-11-23T01:33:55.0419816Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:33:55.0420386Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:33:55.0420854Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:33:55.0421263Z Default Rounding Mode: NEAR 2022-11-23T01:33:55.0421655Z Default Rounding Mode: NEAR 2022-11-23T01:33:55.0422021Z Fast f16: TRUE 2022-11-23T01:33:55.0422383Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:33:55.0422742Z Workgroup Max Size per Dimension: 2022-11-23T01:33:55.0423103Z x 1024(0x400) 2022-11-23T01:33:55.0423447Z y 1024(0x400) 2022-11-23T01:33:55.0423787Z z 1024(0x400) 2022-11-23T01:33:55.0424154Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:33:55.0424541Z Grid Max Size per Dimension: 2022-11-23T01:33:55.0424884Z x 4294967295(0xffffffff) 2022-11-23T01:33:55.0425286Z y 4294967295(0xffffffff) 2022-11-23T01:33:55.0425665Z z 4294967295(0xffffffff) 2022-11-23T01:33:55.0426078Z FBarrier Max Size: 32 2022-11-23T01:33:55.0426402Z *** Done *** 2022-11-23T01:33:55.0446076Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2022-11-23T01:33:55.0446605Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2022-11-23T01:33:55.0446987Z if [[ "x$ngpu" != "x2" && "x$ngpu" != "x4" ]]; then 2022-11-23T01:33:55.0447275Z  if [[ $ngpu -eq 0 ]]; then 2022-11-23T01:33:55.0447597Z  echo "Error: Failed to detect any GPUs on the runner" 2022-11-23T01:33:55.0447891Z  else 2022-11-23T01:33:55.0448228Z  echo "Error: Detected $ngpu GPUs on the runner, when only 2 or 4 were expected" 2022-11-23T01:33:55.0448540Z  fi 2022-11-23T01:33:55.0448953Z  echo "Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2022-11-23T01:33:55.0449357Z  exit 1 2022-11-23T01:33:55.0449575Z fi 2022-11-23T01:33:55.0473476Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:55.0473803Z env: 2022-11-23T01:33:55.0474072Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:55.0474430Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:55.0474749Z ##[endgroup] 2022-11-23T01:33:55.1373010Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:33:55.1374071Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:33:55.1375041Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2022-11-23T01:33:55.1424749Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:55.1425499Z env: 2022-11-23T01:33:55.1426105Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:55.1426898Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:55.1427602Z ##[endgroup] 2022-11-23T01:33:55.1676156Z ##[group]Run # Examine the runner name. If it ends with "-2", this is the second runner on the host. 2022-11-23T01:33:55.1676709Z # Examine the runner name. If it ends with "-2", this is the second runner on the host. 2022-11-23T01:33:55.1677122Z if [[ worker-rocm-amd-90 == *-2 ]]; then 2022-11-23T01:33:55.1677461Z  # select the last two GPUs on the host 2022-11-23T01:33:55.1678009Z  echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD130 --device=/dev/dri/renderD131 --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2022-11-23T01:33:55.1678509Z else 2022-11-23T01:33:55.1678811Z  # select the first two GPUs on the host 2022-11-23T01:33:55.1679476Z  echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2022-11-23T01:33:55.1680035Z fi 2022-11-23T01:33:55.1703366Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:55.1703752Z env: 2022-11-23T01:33:55.1704105Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:55.1704472Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:55.1704833Z ##[endgroup] 2022-11-23T01:33:55.1878291Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2022-11-23T01:33:55.1878684Z with: 2022-11-23T01:33:55.1879197Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:33:55.1879697Z env: 2022-11-23T01:33:55.1879917Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:55.1880216Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:55.1880688Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:33:55.1881095Z ##[endgroup] 2022-11-23T01:33:55.1894679Z ##[group]Run retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:33:55.1895101Z retry () { "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") } 2022-11-23T01:33:55.1895520Z # ignore output since only exit code is used for conditional 2022-11-23T01:33:55.1895994Z # only pull docker image if it's not available locally 2022-11-23T01:33:55.1896583Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2022-11-23T01:33:55.1897017Z  retry docker pull "${DOCKER_IMAGE}" 2022-11-23T01:33:55.1897322Z fi 2022-11-23T01:33:55.1919773Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:55.1920054Z env: 2022-11-23T01:33:55.1920288Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:55.1920588Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:55.1921062Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:33:55.1921680Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:33:55.1922099Z ##[endgroup] 2022-11-23T01:33:55.2492829Z ##[group]Run python3 -m pip install psutil==5.9.1 2022-11-23T01:33:55.2493515Z python3 -m pip install psutil==5.9.1 2022-11-23T01:33:55.2494099Z python3 -m pip install pynvml==11.4.1 2022-11-23T01:33:55.2494745Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2022-11-23T01:33:55.2495442Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2022-11-23T01:33:55.2530758Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:33:55.2531325Z env: 2022-11-23T01:33:55.2531760Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:55.2532321Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:55.2533205Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:33:55.2533953Z ##[endgroup] 2022-11-23T01:33:56.1646358Z Collecting psutil==5.9.1 2022-11-23T01:33:56.4989735Z Installing collected packages: psutil 2022-11-23T01:33:56.6221985Z Successfully installed psutil-5.9.4 2022-11-23T01:33:57.5646090Z Collecting pynvml==11.4.1 2022-11-23T01:33:57.6305193Z Using cached https://files.pythonhosted.org/packages/cc/0a/47be6726fd13f1f4371fa858b506228ed12bc418c07ffcaa6c0f7ceedac0/pynvml-11.4.1-py3-none-any.whl 2022-11-23T01:33:57.6338934Z Installing collected packages: pynvml 2022-11-23T01:33:57.6892797Z Successfully installed pynvml-11.4.1 2022-11-23T01:33:57.7354189Z Prepare all required actions 2022-11-23T01:33:57.7354710Z Getting action download info 2022-11-23T01:33:57.9542125Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea) 2022-11-23T01:33:58.9827503Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7) 2022-11-23T01:33:59.7679308Z ##[group]Run ./.github/actions/download-build-artifacts 2022-11-23T01:33:59.7679639Z with: 2022-11-23T01:33:59.7679908Z name: linux-focal-rocm5.2-py3.8 2022-11-23T01:33:59.7680203Z env: 2022-11-23T01:33:59.7680468Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:59.7680813Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:59.7681327Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:33:59.7681776Z ##[endgroup] 2022-11-23T01:33:59.7715926Z ##[group]Run seemethere/download-artifact-s3@v4 2022-11-23T01:33:59.7716251Z with: 2022-11-23T01:33:59.7716527Z name: linux-focal-rocm5.2-py3.8 2022-11-23T01:33:59.7716833Z s3-bucket: gha-artifacts 2022-11-23T01:33:59.7717127Z region: us-east-1 2022-11-23T01:33:59.7717383Z env: 2022-11-23T01:33:59.7717651Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:33:59.7717986Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:33:59.7718481Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:33:59.7718938Z ##[endgroup] 2022-11-23T01:34:00.4804731Z Found 1 objects with prefix pytorch/pytorch/3528394938/linux-focal-rocm5.2-py3.8/ 2022-11-23T01:34:00.4806854Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-11-23T01:34:13.1808845Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-11-23T01:34:13.1809807Z 2022-11-23T01:34:13.1849950Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ 2022-11-23T01:34:13.1869985Z Artifact download has finished successfully 2022-11-23T01:34:13.2072512Z ##[group]Run unzip -o artifacts.zip 2022-11-23T01:34:13.2073183Z unzip -o artifacts.zip 2022-11-23T01:34:13.2117900Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:13.2118648Z env: 2022-11-23T01:34:13.2119258Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:13.2120059Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:13.2121336Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:13.2122439Z ##[endgroup] 2022-11-23T01:34:13.2197832Z Archive: artifacts.zip 2022-11-23T01:34:13.2199148Z creating: dist/ 2022-11-23T01:34:14.4761125Z inflating: dist/torch-1.14.0a0+git1cfd385-cp38-cp38-linux_x86_64.whl 2022-11-23T01:34:14.4761794Z creating: build/custom_test_artifacts/ 2022-11-23T01:34:14.4762521Z creating: build/custom_test_artifacts/custom-op-build/ 2022-11-23T01:34:14.4763295Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2022-11-23T01:34:14.4766515Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:34:14.4767561Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/ 2022-11-23T01:34:14.4768582Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:34:14.4769527Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:34:14.4771325Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:34:14.4772720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:34:14.4774178Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:34:14.4775224Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:34:14.4776141Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:34:14.4777860Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:34:14.4779634Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:34:14.4781564Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:34:14.4782503Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:34:14.4784230Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:34:14.4786032Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:34:14.4787696Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:34:14.4789214Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2022-11-23T01:34:14.4790779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2022-11-23T01:34:14.4792239Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2022-11-23T01:34:14.4793700Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2022-11-23T01:34:14.4795595Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2022-11-23T01:34:14.4797103Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2022-11-23T01:34:14.4798564Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2022-11-23T01:34:14.4800043Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2022-11-23T01:34:14.4801492Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2022-11-23T01:34:14.4802977Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2022-11-23T01:34:14.4804438Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2022-11-23T01:34:14.4805918Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2022-11-23T01:34:14.4810015Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2022-11-23T01:34:14.4910455Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2022-11-23T01:34:14.4912820Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2022-11-23T01:34:14.4914970Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2022-11-23T01:34:14.4916789Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2022-11-23T01:34:14.4918523Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2022-11-23T01:34:14.4920196Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2022-11-23T01:34:14.4921925Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2022-11-23T01:34:14.4923641Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2022-11-23T01:34:14.4925622Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2022-11-23T01:34:14.4927114Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2022-11-23T01:34:14.4928537Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2022-11-23T01:34:14.4934997Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2022-11-23T01:34:14.5007593Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2022-11-23T01:34:14.5009300Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:34:14.5011280Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:34:14.5012777Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2022-11-23T01:34:14.5014147Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2022-11-23T01:34:14.5015431Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2022-11-23T01:34:14.5016663Z inflating: build/custom_test_artifacts/custom-op-build/detect_rocm_version.cc 2022-11-23T01:34:14.5017844Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2022-11-23T01:34:14.5018963Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2022-11-23T01:34:14.5020123Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2022-11-23T01:34:14.5097772Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2022-11-23T01:34:14.5152217Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2022-11-23T01:34:14.5153606Z creating: build/custom_test_artifacts/jit-hook-build/ 2022-11-23T01:34:14.5155248Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2022-11-23T01:34:14.5156976Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:34:14.5158475Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/ 2022-11-23T01:34:14.5160115Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:34:14.5161912Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:34:14.5163575Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:34:14.5165528Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:34:14.5167207Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:34:14.5168749Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:34:14.5170214Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:34:14.5172277Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:34:14.5174125Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:34:14.5175825Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:34:14.5177612Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:34:14.5179190Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:34:14.5180717Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:34:14.5182063Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:34:14.5183379Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2022-11-23T01:34:14.5184933Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2022-11-23T01:34:14.5186291Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2022-11-23T01:34:14.5187775Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2022-11-23T01:34:14.5189328Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2022-11-23T01:34:14.5190825Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2022-11-23T01:34:14.5192265Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2022-11-23T01:34:14.5193747Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2022-11-23T01:34:14.5195209Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2022-11-23T01:34:14.5196799Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2022-11-23T01:34:14.5198260Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2022-11-23T01:34:14.5199720Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2022-11-23T01:34:14.5201163Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2022-11-23T01:34:14.5254540Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2022-11-23T01:34:14.5256696Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:34:14.5258831Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:34:14.5260708Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2022-11-23T01:34:14.5262353Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2022-11-23T01:34:14.5263949Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2022-11-23T01:34:14.5265532Z inflating: build/custom_test_artifacts/jit-hook-build/detect_rocm_version.cc 2022-11-23T01:34:14.5267040Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2022-11-23T01:34:14.5268447Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2022-11-23T01:34:14.5269898Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2022-11-23T01:34:14.5306834Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2022-11-23T01:34:14.5308218Z creating: build/custom_test_artifacts/custom-backend-build/ 2022-11-23T01:34:14.5309413Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2022-11-23T01:34:14.5310884Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2022-11-23T01:34:14.5312229Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/ 2022-11-23T01:34:14.5313662Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake 2022-11-23T01:34:14.5315060Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/ 2022-11-23T01:34:14.5316582Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/ 2022-11-23T01:34:14.5318025Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c 2022-11-23T01:34:14.5319523Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out 2022-11-23T01:34:14.5320892Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/ 2022-11-23T01:34:14.5322278Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/ 2022-11-23T01:34:14.5323936Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-11-23T01:34:14.5325573Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out 2022-11-23T01:34:14.5327216Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin 2022-11-23T01:34:14.5328812Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake 2022-11-23T01:34:14.5330424Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin 2022-11-23T01:34:14.5335032Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake 2022-11-23T01:34:14.5336692Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2022-11-23T01:34:14.5338278Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2022-11-23T01:34:14.5339889Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2022-11-23T01:34:14.5341485Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2022-11-23T01:34:14.5343207Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2022-11-23T01:34:14.5345009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2022-11-23T01:34:14.5346781Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2022-11-23T01:34:14.5349060Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2022-11-23T01:34:14.5350783Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2022-11-23T01:34:14.5352487Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2022-11-23T01:34:14.5354253Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2022-11-23T01:34:14.5355975Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2022-11-23T01:34:14.5357713Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2022-11-23T01:34:14.5359535Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2022-11-23T01:34:14.5469305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2022-11-23T01:34:14.5471169Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2022-11-23T01:34:14.5472970Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2022-11-23T01:34:14.5474856Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2022-11-23T01:34:14.5476642Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2022-11-23T01:34:14.5478394Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2022-11-23T01:34:14.5480209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2022-11-23T01:34:14.5481983Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2022-11-23T01:34:14.5483786Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2022-11-23T01:34:14.5485557Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2022-11-23T01:34:14.5487635Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2022-11-23T01:34:14.5494374Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2022-11-23T01:34:14.5546412Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2022-11-23T01:34:14.5548346Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:34:14.5550049Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2022-11-23T01:34:14.5551689Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2022-11-23T01:34:14.5553236Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2022-11-23T01:34:14.5554770Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2022-11-23T01:34:14.5556302Z inflating: build/custom_test_artifacts/custom-backend-build/detect_rocm_version.cc 2022-11-23T01:34:14.5557788Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2022-11-23T01:34:14.5559185Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2022-11-23T01:34:14.5560604Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2022-11-23T01:34:14.5659445Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2022-11-23T01:34:14.5699965Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2022-11-23T01:34:14.5701248Z creating: build/lib/ 2022-11-23T01:34:14.5701985Z inflating: build/lib/libclog.a 2022-11-23T01:34:14.5769873Z inflating: build/lib/libgtest.a 2022-11-23T01:34:14.5777882Z inflating: build/lib/libpthreadpool.a 2022-11-23T01:34:14.5855432Z inflating: build/lib/libbenchmark.a 2022-11-23T01:34:14.5947285Z inflating: build/lib/libprotobuf-lite.a 2022-11-23T01:34:14.5953724Z inflating: build/lib/libittnotify.a 2022-11-23T01:34:14.5983231Z inflating: build/lib/libtensorpipe_uv.a 2022-11-23T01:34:14.6048995Z inflating: build/lib/libasmjit.a 2022-11-23T01:34:14.6518756Z inflating: build/lib/libprotobuf.a 2022-11-23T01:34:14.6616381Z inflating: build/lib/libgloo.a 2022-11-23T01:34:14.6645676Z inflating: build/lib/libfmt.a 2022-11-23T01:34:14.6647098Z inflating: build/lib/libcaffe2_nvrtc.so 2022-11-23T01:34:14.6647958Z inflating: build/lib/libfoxi_loader.a 2022-11-23T01:34:14.6711716Z inflating: build/lib/libc10.so 2022-11-23T01:34:14.6712477Z inflating: build/lib/libtorch_global_deps.so 2022-11-23T01:34:14.6721262Z inflating: build/lib/libcpuinfo.a 2022-11-23T01:34:14.6723359Z inflating: build/lib/libnnpack_reference_layers.a 2022-11-23T01:34:14.6731464Z inflating: build/lib/libcpuinfo_internals.a 2022-11-23T01:34:14.6750504Z inflating: build/lib/libgmock.a 2022-11-23T01:34:14.6751626Z inflating: build/lib/libgtest_main.a 2022-11-23T01:34:14.6752523Z inflating: build/lib/libbenchmark_main.a 2022-11-23T01:34:15.5412878Z inflating: build/lib/libdnnl.a 2022-11-23T01:34:15.5932192Z inflating: build/lib/libprotoc.a 2022-11-23T01:34:15.6524791Z inflating: build/lib/libtensorpipe.a 2022-11-23T01:34:15.6927490Z inflating: build/lib/libgloo_hip.a 2022-11-23T01:34:15.6965297Z inflating: build/lib/libc10_hip.so 2022-11-23T01:34:15.6965740Z inflating: build/lib/libgmock_main.a 2022-11-23T01:34:15.8251754Z inflating: build/lib/libfbgemm.a 2022-11-23T01:34:15.8265441Z inflating: build/lib/libqnnpack.a 2022-11-23T01:34:15.9282955Z inflating: build/lib/libdnnl_graph.a 2022-11-23T01:34:15.9581739Z inflating: build/lib/libkineto.a 2022-11-23T01:34:15.9601534Z inflating: build/lib/libpytorch_qnnpack.a 2022-11-23T01:34:15.9642084Z inflating: build/lib/libcaffe2_protos.a 2022-11-23T01:34:15.9762167Z inflating: build/lib/libXNNPACK.a 2022-11-23T01:34:15.9804389Z inflating: build/lib/libonnx_proto.a 2022-11-23T01:34:16.0403672Z inflating: build/lib/libonnx.a 2022-11-23T01:34:16.0422401Z inflating: build/lib/libnnpack.a 2022-11-23T01:34:18.4504244Z inflating: build/lib/libtorch_cpu.so 2022-11-23T01:34:19.6242347Z inflating: build/lib/libtorch_hip.so 2022-11-23T01:34:19.6243184Z inflating: build/lib/libtorch.so 2022-11-23T01:34:19.6269344Z inflating: build/lib/libjitbackend_test.so 2022-11-23T01:34:19.6325668Z inflating: build/lib/libtorchbind_test.so 2022-11-23T01:34:19.6355631Z inflating: build/lib/libbackend_with_compiler.so 2022-11-23T01:34:19.6357000Z inflating: build/lib/libshm.so 2022-11-23T01:34:19.8120811Z inflating: build/lib/libtorch_python.so 2022-11-23T01:34:19.8159193Z inflating: build/lib/libnnapi_backend.so 2022-11-23T01:34:19.8160322Z creating: build/bin/ 2022-11-23T01:34:19.8161044Z creating: build/bin/CMakeFiles/ 2022-11-23T01:34:19.8161993Z inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake 2022-11-23T01:34:19.8163019Z extracting: build/bin/CMakeFiles/progress.marks 2022-11-23T01:34:19.8214316Z inflating: build/bin/hip_generator_test 2022-11-23T01:34:19.8215156Z inflating: build/bin/Makefile 2022-11-23T01:34:19.8266588Z inflating: build/bin/variant_test 2022-11-23T01:34:19.8267456Z inflating: build/bin/cmake_install.cmake 2022-11-23T01:34:19.8321994Z inflating: build/bin/undefined_tensor_test 2022-11-23T01:34:19.8322898Z inflating: build/bin/CTestTestfile.cmake 2022-11-23T01:34:19.8376027Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2022-11-23T01:34:19.8430484Z inflating: build/bin/c10_DeviceGuard_test 2022-11-23T01:34:19.8484485Z inflating: build/bin/c10_Device_test 2022-11-23T01:34:19.8547033Z inflating: build/bin/c10_DispatchKeySet_test 2022-11-23T01:34:19.8598185Z inflating: build/bin/c10_StreamGuard_test 2022-11-23T01:34:19.8650514Z inflating: build/bin/c10_SymInt_test 2022-11-23T01:34:19.8708854Z inflating: build/bin/c10_InlineDeviceGuard_test 2022-11-23T01:34:19.8767364Z inflating: build/bin/c10_InlineStreamGuard_test 2022-11-23T01:34:19.8826350Z inflating: build/bin/c10_SizesAndStrides_test 2022-11-23T01:34:19.8878082Z inflating: build/bin/c10_Array_test 2022-11-23T01:34:19.8933278Z inflating: build/bin/c10_Bitset_test 2022-11-23T01:34:19.8985595Z inflating: build/bin/c10_ConstexprCrc_test 2022-11-23T01:34:19.9040008Z inflating: build/bin/c10_C++17_test 2022-11-23T01:34:19.9092300Z inflating: build/bin/c10_DeadlockDetection_test 2022-11-23T01:34:19.9145244Z inflating: build/bin/c10_Half_test 2022-11-23T01:34:19.9204746Z inflating: build/bin/c10_LeftRight_test 2022-11-23T01:34:19.9272753Z inflating: build/bin/c10_Metaprogramming_test 2022-11-23T01:34:19.9326810Z inflating: build/bin/c10_Synchronized_test 2022-11-23T01:34:19.9485236Z inflating: build/bin/c10_SmallVectorTest 2022-11-23T01:34:19.9546409Z inflating: build/bin/c10_ThreadLocal_test 2022-11-23T01:34:19.9601081Z inflating: build/bin/c10_TypeIndex_test 2022-11-23T01:34:19.9652158Z inflating: build/bin/c10_TypeTraits_test 2022-11-23T01:34:19.9706975Z inflating: build/bin/c10_TypeList_test 2022-11-23T01:34:19.9761540Z inflating: build/bin/c10_accumulate_test 2022-11-23T01:34:19.9819546Z inflating: build/bin/c10_bfloat16_test 2022-11-23T01:34:19.9877069Z inflating: build/bin/c10_complex_math_test 2022-11-23T01:34:19.9930433Z inflating: build/bin/c10_flags_test 2022-11-23T01:34:19.9985860Z inflating: build/bin/c10_exception_test 2022-11-23T01:34:20.0044718Z inflating: build/bin/c10_complex_test 2022-11-23T01:34:20.0163268Z inflating: build/bin/c10_either_test 2022-11-23T01:34:20.0215376Z inflating: build/bin/c10_irange_test 2022-11-23T01:34:20.0277065Z inflating: build/bin/c10_logging_test 2022-11-23T01:34:20.0451393Z inflating: build/bin/c10_intrusive_ptr_test 2022-11-23T01:34:20.0526422Z inflating: build/bin/c10_optional_test 2022-11-23T01:34:20.0586064Z inflating: build/bin/c10_registry_test 2022-11-23T01:34:20.0653971Z inflating: build/bin/c10_ordered_preserving_dict_test 2022-11-23T01:34:20.0706235Z inflating: build/bin/c10_tempfile_test 2022-11-23T01:34:20.0769629Z inflating: build/bin/c10_string_view_test 2022-11-23T01:34:20.0829086Z inflating: build/bin/c10_typeid_test 2022-11-23T01:34:20.0881375Z inflating: build/bin/c10_intrusive_ptr_benchmark 2022-11-23T01:34:20.1346352Z inflating: build/bin/protoc-3.13.0.0 2022-11-23T01:34:20.1811211Z inflating: build/bin/protoc 2022-11-23T01:34:20.1863298Z inflating: build/bin/c10_hip_HIPTest 2022-11-23T01:34:20.2165617Z inflating: build/bin/vec_test_all_types_DEFAULT 2022-11-23T01:34:20.2489930Z inflating: build/bin/vec_test_all_types_AVX512 2022-11-23T01:34:20.2826465Z inflating: build/bin/vec_test_all_types_AVX2 2022-11-23T01:34:20.2880319Z inflating: build/bin/HashStoreTest 2022-11-23T01:34:20.2935257Z inflating: build/bin/FileStoreTest 2022-11-23T01:34:20.2998131Z inflating: build/bin/TCPStoreTest 2022-11-23T01:34:20.3000114Z inflating: build/bin/example_allreduce 2022-11-23T01:34:20.3070093Z inflating: build/bin/ProcessGroupGlooTest 2022-11-23T01:34:20.3125980Z inflating: build/bin/Dimname_test 2022-11-23T01:34:20.3202584Z inflating: build/bin/Dict_test 2022-11-23T01:34:20.3263014Z inflating: build/bin/NamedTensor_test 2022-11-23T01:34:20.3331083Z inflating: build/bin/MaybeOwned_test 2022-11-23T01:34:20.3395533Z inflating: build/bin/static_runtime_bench 2022-11-23T01:34:20.3457118Z inflating: build/bin/apply_utils_test 2022-11-23T01:34:20.3522238Z inflating: build/bin/basic 2022-11-23T01:34:20.3584381Z inflating: build/bin/atest 2022-11-23T01:34:20.3642243Z inflating: build/bin/broadcast_test 2022-11-23T01:34:20.3704064Z inflating: build/bin/cpu_generator_test 2022-11-23T01:34:20.3760171Z inflating: build/bin/cpu_profiling_allocator_test 2022-11-23T01:34:20.4028994Z inflating: build/bin/static_runtime_test 2022-11-23T01:34:20.4083175Z inflating: build/bin/dispatch_key_set_test 2022-11-23T01:34:20.4177507Z inflating: build/bin/cpu_rng_test 2022-11-23T01:34:20.4233372Z inflating: build/bin/dlconvertor_test 2022-11-23T01:34:20.4295152Z inflating: build/bin/extension_backend_test 2022-11-23T01:34:20.4353271Z inflating: build/bin/half_test 2022-11-23T01:34:20.4406525Z inflating: build/bin/lazy_tensor_test 2022-11-23T01:34:20.4462981Z inflating: build/bin/math_kernel_test 2022-11-23T01:34:20.4563695Z inflating: build/bin/ivalue_test 2022-11-23T01:34:20.4620966Z inflating: build/bin/memory_format_test 2022-11-23T01:34:20.4676137Z inflating: build/bin/memory_overlapping_test 2022-11-23T01:34:20.4729869Z inflating: build/bin/operator_name_test 2022-11-23T01:34:20.4785381Z inflating: build/bin/mobile_memory_cleanup 2022-11-23T01:34:20.4845282Z inflating: build/bin/native_test 2022-11-23T01:34:20.4899948Z inflating: build/bin/operators_test 2022-11-23T01:34:20.4954381Z inflating: build/bin/packedtensoraccessor_test 2022-11-23T01:34:20.5014773Z inflating: build/bin/quantized_test 2022-11-23T01:34:20.5068173Z inflating: build/bin/reduce_ops_test 2022-11-23T01:34:20.5121975Z inflating: build/bin/reportMemoryUsage_test 2022-11-23T01:34:20.5190443Z inflating: build/bin/pow_test 2022-11-23T01:34:20.5249794Z inflating: build/bin/scalar_tensor_test 2022-11-23T01:34:20.5310301Z inflating: build/bin/scalar_test 2022-11-23T01:34:20.5365117Z inflating: build/bin/stride_properties_test 2022-11-23T01:34:20.5367253Z inflating: build/bin/thread_init_test 2022-11-23T01:34:20.5427745Z inflating: build/bin/type_ptr_test 2022-11-23T01:34:20.5509771Z inflating: build/bin/tensor_iterator_test 2022-11-23T01:34:20.5567090Z inflating: build/bin/test_parallel 2022-11-23T01:34:20.5631382Z inflating: build/bin/type_test 2022-11-23T01:34:20.5632284Z inflating: build/bin/verify_api_visibility 2022-11-23T01:34:20.5706718Z inflating: build/bin/vmap_test 2022-11-23T01:34:20.5761300Z inflating: build/bin/weakref_test 2022-11-23T01:34:20.5816408Z inflating: build/bin/wrapdim_test 2022-11-23T01:34:20.5868301Z inflating: build/bin/xla_tensor_test 2022-11-23T01:34:20.5980230Z inflating: build/bin/List_test 2022-11-23T01:34:20.6043149Z inflating: build/bin/IListRef_test 2022-11-23T01:34:20.6169955Z inflating: build/bin/kernel_function_legacy_test 2022-11-23T01:34:20.6238749Z inflating: build/bin/KernelFunction_test 2022-11-23T01:34:20.6338034Z inflating: build/bin/kernel_function_test 2022-11-23T01:34:20.6470248Z inflating: build/bin/kernel_lambda_legacy_test 2022-11-23T01:34:20.6578337Z inflating: build/bin/kernel_lambda_test 2022-11-23T01:34:20.6641106Z inflating: build/bin/kernel_stackbased_test 2022-11-23T01:34:20.6740577Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2022-11-23T01:34:20.6794431Z inflating: build/bin/CppSignature_test 2022-11-23T01:34:20.6847028Z inflating: build/bin/op_allowlist_test 2022-11-23T01:34:20.6906451Z inflating: build/bin/backend_fallback_test 2022-11-23T01:34:20.6958817Z inflating: build/bin/hip_complex_math_test 2022-11-23T01:34:20.7014416Z inflating: build/bin/inline_container_test 2022-11-23T01:34:20.7321316Z inflating: build/bin/op_registration_test 2022-11-23T01:34:20.7373377Z inflating: build/bin/hip_complex_test 2022-11-23T01:34:20.7429027Z inflating: build/bin/hip_apply_test 2022-11-23T01:34:20.7480746Z inflating: build/bin/hip_distributions_test 2022-11-23T01:34:20.7532578Z inflating: build/bin/hip_half_test 2022-11-23T01:34:20.7583847Z inflating: build/bin/hip_integer_divider_test 2022-11-23T01:34:20.7635153Z inflating: build/bin/hip_optional_test 2022-11-23T01:34:20.7686086Z inflating: build/bin/hip_packedtensoraccessor_test 2022-11-23T01:34:20.7741073Z inflating: build/bin/hip_dlconvertor_test 2022-11-23T01:34:20.7792722Z inflating: build/bin/hip_vectorized_test 2022-11-23T01:34:20.7810788Z inflating: build/bin/tutorial_tensorexpr 2022-11-23T01:34:20.7872032Z inflating: build/bin/test_dist_autograd 2022-11-23T01:34:20.7948158Z inflating: build/bin/test_cpp_rpc 2022-11-23T01:34:20.8777146Z inflating: build/bin/test_tensorexpr 2022-11-23T01:34:20.8778223Z inflating: build/bin/parallel_benchmark 2022-11-23T01:34:20.8849286Z inflating: build/bin/test_mobile_nnc 2022-11-23T01:34:20.8860758Z inflating: build/bin/aot_model_compiler_test 2022-11-23T01:34:20.9223982Z inflating: build/bin/test_lazy 2022-11-23T01:34:21.0401812Z inflating: build/bin/test_api 2022-11-23T01:34:21.0406154Z inflating: build/bin/torch_shm_manager 2022-11-23T01:34:21.0988816Z inflating: build/bin/test_jit 2022-11-23T01:34:21.0990074Z inflating: .pytorch-test-times.json 2022-11-23T01:34:21.1036731Z ##[group]Run df -H 2022-11-23T01:34:21.1037270Z df -H 2022-11-23T01:34:21.1078976Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T01:34:21.1079601Z env: 2022-11-23T01:34:21.1080112Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:21.1080785Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:21.1081821Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:21.1082754Z ##[endgroup] 2022-11-23T01:34:21.1156569Z Filesystem Size Used Avail Use% Mounted on 2022-11-23T01:34:21.1157467Z udev 136G 0 136G 0% /dev 2022-11-23T01:34:21.1158216Z tmpfs 28G 1.6M 28G 1% /run 2022-11-23T01:34:21.1158930Z /dev/nvme0n1p1 984G 383G 551G 42% / 2022-11-23T01:34:21.1159954Z tmpfs 136G 8.2k 136G 1% /dev/shm 2022-11-23T01:34:21.1160682Z tmpfs 5.3M 0 5.3M 0% /run/lock 2022-11-23T01:34:21.1161745Z tmpfs 136G 0 136G 0% /sys/fs/cgroup 2022-11-23T01:34:21.1162550Z tmpfs 28G 13k 28G 1% /run/user/1121 2022-11-23T01:34:21.1163314Z tmpfs 28G 0 28G 0% /run/user/1000 2022-11-23T01:34:21.1205662Z ##[group]Run .github/scripts/parse_ref.py 2022-11-23T01:34:21.1206395Z .github/scripts/parse_ref.py 2022-11-23T01:34:21.1246941Z shell: /bin/bash -e {0} 2022-11-23T01:34:21.1247414Z env: 2022-11-23T01:34:21.1247881Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:21.1248483Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:21.1249435Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:21.1250276Z ##[endgroup] 2022-11-23T01:34:21.1648404Z ##[group]Run set -x 2022-11-23T01:34:21.1649251Z set -x 2022-11-23T01:34:21.1649830Z  2022-11-23T01:34:21.1650695Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2022-11-23T01:34:21.1651613Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2022-11-23T01:34:21.1652561Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2022-11-23T01:34:21.1653432Z  TEST_COMMAND=.jenkins/caffe2/test.sh 2022-11-23T01:34:21.1654141Z else 2022-11-23T01:34:21.1654876Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2022-11-23T01:34:21.1655596Z fi 2022-11-23T01:34:21.1656138Z  2022-11-23T01:34:21.1656990Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}") 2022-11-23T01:34:21.1657832Z  2022-11-23T01:34:21.1658606Z # sanitize the input commit message and PR body here: 2022-11-23T01:34:21.1659668Z # 2022-11-23T01:34:21.1660620Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment 2022-11-23T01:34:21.1661932Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028 2022-11-23T01:34:21.1663070Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}" 2022-11-23T01:34:21.1663901Z PR_BODY="${PR_BODY//[$'\n\r']}" 2022-11-23T01:34:21.1664570Z  2022-11-23T01:34:21.1665506Z # then trim all special characters like single and double quotes to avoid unescaped inputs to 2022-11-23T01:34:21.1666503Z # wreak havoc internally 2022-11-23T01:34:21.1667356Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}" 2022-11-23T01:34:21.1668228Z export PR_BODY="${PR_BODY//[\'\"]}" 2022-11-23T01:34:21.1668896Z  2022-11-23T01:34:21.1669724Z # detached container should get cleaned up by teardown_ec2_linux 2022-11-23T01:34:21.1670818Z # TODO: Stop building test binaries as part of the build phase 2022-11-23T01:34:21.1671814Z # Used for GPU_FLAG since that doesn't play nice 2022-11-23T01:34:21.1672684Z # shellcheck disable=SC2086,SC2090 2022-11-23T01:34:21.1673450Z container_name=$(docker run \ 2022-11-23T01:34:21.1674174Z  ${GPU_FLAG:-} \ 2022-11-23T01:34:21.1674887Z  -e BUILD_ENVIRONMENT \ 2022-11-23T01:34:21.1675600Z  -e PR_NUMBER \ 2022-11-23T01:34:21.1676301Z  -e GITHUB_ACTIONS \ 2022-11-23T01:34:21.1676981Z  -e BRANCH \ 2022-11-23T01:34:21.1677598Z  -e SHA1 \ 2022-11-23T01:34:21.1678284Z  -e AWS_DEFAULT_REGION \ 2022-11-23T01:34:21.1679009Z  -e IN_WHEEL_TEST \ 2022-11-23T01:34:21.1679699Z  -e SHARD_NUMBER \ 2022-11-23T01:34:21.1680387Z  -e TEST_CONFIG \ 2022-11-23T01:34:21.1681061Z  -e NUM_TEST_SHARDS \ 2022-11-23T01:34:21.1681741Z  -e PR_BODY \ 2022-11-23T01:34:21.1682440Z  -e COMMIT_MESSAGES \ 2022-11-23T01:34:21.1683216Z  -e PYTORCH_RETRY_TEST_CASES \ 2022-11-23T01:34:21.1684048Z  -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \ 2022-11-23T01:34:21.1684892Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2022-11-23T01:34:21.1685640Z  -e SCCACHE_BUCKET \ 2022-11-23T01:34:21.1686421Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2022-11-23T01:34:21.1687283Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2022-11-23T01:34:21.1688164Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2022-11-23T01:34:21.1689097Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2022-11-23T01:34:21.1689945Z  --ulimit stack=10485760:83886080 \ 2022-11-23T01:34:21.1690982Z  --security-opt seccomp=unconfined \ 2022-11-23T01:34:21.1691783Z  --cap-add=SYS_PTRACE \ 2022-11-23T01:34:21.1692504Z  --shm-size="8g" \ 2022-11-23T01:34:21.1693151Z  --tty \ 2022-11-23T01:34:21.1693791Z  --detach \ 2022-11-23T01:34:21.1694717Z  --name="${container_name}" \ 2022-11-23T01:34:21.1695439Z  --user jenkins \ 2022-11-23T01:34:21.1696306Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2022-11-23T01:34:21.1697227Z  -w /var/lib/jenkins/workspace \ 2022-11-23T01:34:21.1697967Z  "${DOCKER_IMAGE}" 2022-11-23T01:34:21.1698590Z ) 2022-11-23T01:34:21.1699286Z # save container name for later step 2022-11-23T01:34:21.1700213Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2022-11-23T01:34:21.1701447Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2022-11-23T01:34:21.1702972Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2022-11-23T01:34:21.1747460Z shell: /bin/bash -e {0} 2022-11-23T01:34:21.1747754Z env: 2022-11-23T01:34:21.1747996Z GIT_DEFAULT_BRANCH: master 2022-11-23T01:34:21.1748310Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T01:34:21.1748787Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T01:34:21.1749262Z BUILD_ENVIRONMENT: linux-focal-rocm5.2-py3.8 2022-11-23T01:34:21.1749552Z PR_NUMBER: 2022-11-23T01:34:21.1749780Z BRANCH: master 2022-11-23T01:34:21.1750055Z SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:34:21.1750349Z PYTORCH_RETRY_TEST_CASES: 1 2022-11-23T01:34:21.1750635Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-11-23T01:34:21.1750912Z TEST_CONFIG: distributed 2022-11-23T01:34:21.1751170Z SHARD_NUMBER: 2 2022-11-23T01:34:21.1751393Z NUM_TEST_SHARDS: 2 2022-11-23T01:34:21.1751632Z PR_BODY: 2022-11-23T01:34:21.1751933Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-11-23T01:34:21.1752444Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:34:21.1752985Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-11-23T01:34:21.1753358Z PYTORCH_JIT_ENABLE_NVFUSER: 1 2022-11-23T01:34:21.1753637Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 1 2022-11-23T01:34:21.1753942Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2022-11-23T01:34:21.1754213Z ##[endgroup] 2022-11-23T01:34:21.1799345Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2022-11-23T01:34:21.1800620Z + [[ linux-focal-rocm5.2-py3.8 == *onnx* ]] 2022-11-23T01:34:21.1801439Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2022-11-23T01:34:21.1805695Z ++ git cherry -v origin/master 2022-11-23T01:34:21.1838674Z + COMMIT_MESSAGES= 2022-11-23T01:34:21.1839554Z + COMMIT_MESSAGES= 2022-11-23T01:34:21.1840199Z + PR_BODY= 2022-11-23T01:34:21.1840859Z + export COMMIT_MESSAGES= 2022-11-23T01:34:21.1841542Z + COMMIT_MESSAGES= 2022-11-23T01:34:21.1842174Z + export PR_BODY= 2022-11-23T01:34:21.1842815Z + PR_BODY= 2022-11-23T01:34:21.1855057Z +++ nproc --ignore=2 2022-11-23T01:34:21.1881659Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e MAX_JOBS=62 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528394938 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940 2022-11-23T01:34:22.9568346Z + container_name=8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T01:34:22.9569941Z + echo CONTAINER_NAME=8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T01:34:22.9572697Z + docker exec -t 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .jenkins/pytorch/test.sh' 2022-11-23T01:34:28.9207428Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp38-cp38-linux_x86_64.whl 2022-11-23T01:34:29.1485903Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (4.4.0) 2022-11-23T01:34:29.1487835Z Requirement already satisfied: networkx in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3) 2022-11-23T01:34:29.1493513Z Requirement already satisfied: sympy in /opt/conda/lib/python3.8/site-packages (from torch==1.14.0a0+git1cfd385) (1.11.1) 2022-11-23T01:34:29.1697116Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1) 2022-11-23T01:34:30.0692571Z Installing collected packages: torch 2022-11-23T01:34:37.0090162Z Successfully installed torch-1.14.0a0+git1cfd385 2022-11-23T01:34:37.0864199Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2022-11-23T01:34:37.1145406Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.8/site-packages/torch 2022-11-23T01:34:37.1147163Z + TORCH_BIN_DIR=/opt/conda/lib/python3.8/site-packages/torch/bin 2022-11-23T01:34:37.1148477Z + TORCH_LIB_DIR=/opt/conda/lib/python3.8/site-packages/torch/lib 2022-11-23T01:34:37.1149745Z + TORCH_TEST_DIR=/opt/conda/lib/python3.8/site-packages/torch/test 2022-11-23T01:34:37.1150556Z + BUILD_DIR=build 2022-11-23T01:34:37.1151225Z + BUILD_RENAMED_DIR=build_renamed 2022-11-23T01:34:37.1151912Z + BUILD_BIN_DIR=build/bin 2022-11-23T01:34:37.1152574Z + export VALGRIND=ON 2022-11-23T01:34:37.1153231Z + VALGRIND=ON 2022-11-23T01:34:37.1154174Z + [[ linux-focal-rocm5.2-py3.8 == *clang9* ]] 2022-11-23T01:34:37.1155219Z + [[ linux-focal-rocm5.2-py3.8 != *bazel* ]] 2022-11-23T01:34:37.1156031Z ++ realpath build/custom_test_artifacts 2022-11-23T01:34:37.1174661Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2022-11-23T01:34:37.1175707Z ++ dirname .jenkins/pytorch/test.sh 2022-11-23T01:34:37.1195176Z + source .jenkins/pytorch/common.sh 2022-11-23T01:34:37.1200851Z +++ dirname .jenkins/pytorch/common.sh 2022-11-23T01:34:37.1218000Z ++ source .jenkins/pytorch/common_utils.sh 2022-11-23T01:34:37.1220932Z +++ declare -f -t trap_add 2022-11-23T01:34:37.1237371Z ++ set -ex 2022-11-23T01:34:37.1238377Z ++ [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:34:37.1239509Z ++ unset HIP_PLATFORM 2022-11-23T01:34:37.1240223Z ++ export PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:34:37.1240952Z ++ PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:34:37.1241709Z ++ export HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:34:37.1242397Z ++ HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:34:37.1243101Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:34:37.1243853Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:34:37.1244544Z ++ BUILD_TEST_LIBTORCH=0 2022-11-23T01:34:37.1245430Z + echo 'Environment variables' 2022-11-23T01:34:37.1246119Z Environment variables 2022-11-23T01:34:37.1246767Z + env 2022-11-23T01:34:37.1262061Z INSTALLED_DB=yes 2022-11-23T01:34:37.1263008Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T01:34:37.1264192Z BUILD_ENVIRONMENT=linux-focal-rocm5.2-py3.8 2022-11-23T01:34:37.1264779Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1 2022-11-23T01:34:37.1265268Z HOSTNAME=8ba7ada97cfc 2022-11-23T01:34:37.1266301Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_4cc7e0ac-285d-4e75-8ac3-f49d948251da 2022-11-23T01:34:37.1267053Z GITHUB_ACTION=__self 2022-11-23T01:34:37.1267540Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 2022-11-23T01:34:37.1268057Z GITHUB_RUN_NUMBER=3445 2022-11-23T01:34:37.1268845Z TEST_CONFIG=distributed 2022-11-23T01:34:37.1269391Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2022-11-23T01:34:37.1269918Z GITHUB_REF_TYPE=branch 2022-11-23T01:34:37.1271689Z *** 2022-11-23T01:34:37.1272094Z GITHUB_ACTIONS=true 2022-11-23T01:34:37.1272612Z SHA1=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:34:37.1273208Z GITHUB_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T01:34:37.1273767Z GITHUB_REF=refs/heads/master 2022-11-23T01:34:37.1274238Z SHARD_NUMBER=2 2022-11-23T01:34:37.1274851Z GITHUB_REF_PROTECTED=true 2022-11-23T01:34:37.1275314Z HOME=/var/lib/jenkins 2022-11-23T01:34:37.1275856Z GITHUB_API_URL=https://api.github.com 2022-11-23T01:34:37.1276428Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2022-11-23T01:34:37.1277003Z LANG=C.UTF-8 2022-11-23T01:34:37.1277451Z PYTORCH_TEST_WITH_ROCM=1 2022-11-23T01:34:37.1277916Z NUM_TEST_SHARDS=2 2022-11-23T01:34:37.1278924Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_4cc7e0ac-285d-4e75-8ac3-f49d948251da 2022-11-23T01:34:37.1279904Z MAGMA_HOME=/opt/rocm/magma 2022-11-23T01:34:37.1280404Z PYTORCH_RETRY_TEST_CASES=1 2022-11-23T01:34:37.1281448Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_4cc7e0ac-285d-4e75-8ac3-f49d948251da 2022-11-23T01:34:37.1282169Z HSAKMT_DEBUG_LEVEL=4 2022-11-23T01:34:37.1283037Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2022-11-23T01:34:37.1283694Z GITHUB_EVENT_NAME=schedule 2022-11-23T01:34:37.1284165Z GITHUB_RUN_ID=3528394938 2022-11-23T01:34:37.1285251Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_4cc7e0ac-285d-4e75-8ac3-f49d948251da 2022-11-23T01:34:37.1286054Z GITHUB_ACTOR=pytorchmergebot 2022-11-23T01:34:37.1286525Z PR_NUMBER= 2022-11-23T01:34:37.1286955Z GITHUB_RUN_ATTEMPT=1 2022-11-23T01:34:37.1287381Z VALGRIND=ON 2022-11-23T01:34:37.1287936Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2022-11-23T01:34:37.1288492Z TERM=xterm 2022-11-23T01:34:37.1288924Z INSTALLED_VISION=yes 2022-11-23T01:34:37.1289365Z BRANCH=master 2022-11-23T01:34:37.1290268Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2022-11-23T01:34:37.1291171Z GITHUB_SERVER_URL=https://github.com 2022-11-23T01:34:37.1291709Z PYTORCH_ROCM_ARCH=gfx906 2022-11-23T01:34:37.1292156Z SHLVL=1 2022-11-23T01:34:37.1292544Z MAX_JOBS=62 2022-11-23T01:34:37.1292961Z COMMIT_MESSAGES= 2022-11-23T01:34:37.1293396Z GITHUB_REF_NAME=master 2022-11-23T01:34:37.1294241Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2022-11-23T01:34:37.1294868Z GITHUB_JOB=test 2022-11-23T01:34:37.1295365Z GITHUB_REPOSITORY=pytorch/pytorch 2022-11-23T01:34:37.1295903Z LC_ALL=C.UTF-8 2022-11-23T01:34:37.1296344Z GITHUB_RETENTION_DAYS=90 2022-11-23T01:34:37.1296834Z GITHUB_ACTION_REPOSITORY= 2022-11-23T01:34:37.1297721Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:34:37.1298530Z GITHUB_BASE_REF= 2022-11-23T01:34:37.1298991Z CI=true 2022-11-23T01:34:37.1299417Z HSA_FORCE_FINE_GRAIN_PCIE=1 2022-11-23T01:34:37.1299932Z GITHUB_REPOSITORY_OWNER=pytorch 2022-11-23T01:34:37.1300418Z INSTALLED_PROTOBUF=yes 2022-11-23T01:34:37.1300865Z GITHUB_HEAD_REF= 2022-11-23T01:34:37.1301292Z GITHUB_ACTION_REF= 2022-11-23T01:34:37.1301993Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2022-11-23T01:34:37.1302567Z GITHUB_WORKFLOW=periodic 2022-11-23T01:34:37.1303059Z DEBIAN_FRONTEND=noninteractive 2022-11-23T01:34:37.1304120Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_4cc7e0ac-285d-4e75-8ac3-f49d948251da 2022-11-23T01:34:37.1304844Z OLDPWD=/var/lib/jenkins 2022-11-23T01:34:37.1305280Z PR_BODY= 2022-11-23T01:34:37.1305673Z _=/usr/bin/env 2022-11-23T01:34:37.1306201Z + echo 'Testing pytorch' 2022-11-23T01:34:37.1306643Z Testing pytorch 2022-11-23T01:34:37.1307133Z + export LANG=C.UTF-8 2022-11-23T01:34:37.1307738Z + LANG=C.UTF-8 2022-11-23T01:34:37.1308158Z + PR_NUMBER= 2022-11-23T01:34:37.1308611Z + [[ distributed == \d\e\f\a\u\l\t ]] 2022-11-23T01:34:37.1309144Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2022-11-23T01:34:37.1309858Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:34:37.1310421Z + export HIP_VISIBLE_DEVICES=0,1 2022-11-23T01:34:37.1310925Z + HIP_VISIBLE_DEVICES=0,1 2022-11-23T01:34:37.1311407Z + [[ distributed == \s\l\o\w ]] 2022-11-23T01:34:37.1312132Z + [[ linux-focal-rocm5.2-py3.8 == *slow-gradcheck* ]] 2022-11-23T01:34:37.1312863Z + [[ linux-focal-rocm5.2-py3.8 == *cuda* ]] 2022-11-23T01:34:37.1313579Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:34:37.1314181Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2022-11-23T01:34:37.1314758Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2022-11-23T01:34:37.1315289Z + [[ distributed == *crossref* ]] 2022-11-23T01:34:37.1315786Z + [[ distributed == *dynamo* ]] 2022-11-23T01:34:37.1316391Z + [[ distributed == *inductor* ]] 2022-11-23T01:34:37.1317080Z + [[ linux-focal-rocm5.2-py3.8 == *rocm* ]] 2022-11-23T01:34:37.1317562Z + rocminfo 2022-11-23T01:34:37.1508627Z ROCk module is loaded 2022-11-23T01:34:37.2146217Z ===================== 2022-11-23T01:34:37.2146740Z HSA System Attributes 2022-11-23T01:34:37.2147187Z ===================== 2022-11-23T01:34:37.2147660Z Runtime Version: 1.1 2022-11-23T01:34:37.2148183Z System Timestamp Freq.: 1000.000000MHz 2022-11-23T01:34:37.2148866Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2022-11-23T01:34:37.2149569Z Machine Model: LARGE 2022-11-23T01:34:37.2150167Z System Endianness: LITTLE 2022-11-23T01:34:37.2150530Z 2022-11-23T01:34:37.2150684Z ========== 2022-11-23T01:34:37.2151260Z HSA Agents 2022-11-23T01:34:37.2151687Z ========== 2022-11-23T01:34:37.2152128Z ******* 2022-11-23T01:34:37.2152561Z Agent 1 2022-11-23T01:34:37.2153041Z ******* 2022-11-23T01:34:37.2153826Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2154580Z Uuid: CPU-XX 2022-11-23T01:34:37.2155381Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2156012Z Vendor Name: CPU 2022-11-23T01:34:37.2156635Z Feature: None specified 2022-11-23T01:34:37.2157414Z Profile: FULL_PROFILE 2022-11-23T01:34:37.2158254Z Float Round Mode: NEAR 2022-11-23T01:34:37.2159065Z Max Queue Number: 0(0x0) 2022-11-23T01:34:37.2159858Z Queue Min Size: 0(0x0) 2022-11-23T01:34:37.2160653Z Queue Max Size: 0(0x0) 2022-11-23T01:34:37.2161463Z Queue Type: MULTI 2022-11-23T01:34:37.2162234Z Node: 0 2022-11-23T01:34:37.2163010Z Device Type: CPU 2022-11-23T01:34:37.2163714Z Cache Info: 2022-11-23T01:34:37.2164418Z L1: 32768(0x8000) KB 2022-11-23T01:34:37.2165174Z Chip ID: 0(0x0) 2022-11-23T01:34:37.2165954Z Cacheline Size: 64(0x40) 2022-11-23T01:34:37.2166755Z Max Clock Freq. (MHz): 2200 2022-11-23T01:34:37.2167532Z BDFID: 0 2022-11-23T01:34:37.2168309Z Internal Node ID: 0 2022-11-23T01:34:37.2169072Z Compute Unit: 16 2022-11-23T01:34:37.2170216Z SIMDs per CU: 0 2022-11-23T01:34:37.2171223Z Shader Engines: 0 2022-11-23T01:34:37.2172031Z Shader Arrs. per Eng.: 0 2022-11-23T01:34:37.2172866Z WatchPts on Addr. Ranges:1 2022-11-23T01:34:37.2173601Z Features: None 2022-11-23T01:34:37.2174259Z Pool Info: 2022-11-23T01:34:37.2174887Z Pool 1 2022-11-23T01:34:37.2175681Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:34:37.2176536Z Size: 65894484(0x3ed7854) KB 2022-11-23T01:34:37.2177344Z Allocatable: TRUE 2022-11-23T01:34:37.2178164Z Alloc Granule: 4KB 2022-11-23T01:34:37.2178982Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2179952Z Accessible by all: TRUE 2022-11-23T01:34:37.2180676Z Pool 2 2022-11-23T01:34:37.2181479Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:34:37.2182342Z Size: 65894484(0x3ed7854) KB 2022-11-23T01:34:37.2183153Z Allocatable: TRUE 2022-11-23T01:34:37.2183969Z Alloc Granule: 4KB 2022-11-23T01:34:37.2184774Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2185605Z Accessible by all: TRUE 2022-11-23T01:34:37.2186330Z Pool 3 2022-11-23T01:34:37.2187119Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:37.2187965Z Size: 65894484(0x3ed7854) KB 2022-11-23T01:34:37.2188780Z Allocatable: TRUE 2022-11-23T01:34:37.2189584Z Alloc Granule: 4KB 2022-11-23T01:34:37.2190400Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2191228Z Accessible by all: TRUE 2022-11-23T01:34:37.2191949Z ISA Info: 2022-11-23T01:34:37.2192565Z ******* 2022-11-23T01:34:37.2193163Z Agent 2 2022-11-23T01:34:37.2193739Z ******* 2022-11-23T01:34:37.2194755Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2195808Z Uuid: CPU-XX 2022-11-23T01:34:37.2196895Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2197758Z Vendor Name: CPU 2022-11-23T01:34:37.2198581Z Feature: None specified 2022-11-23T01:34:37.2199418Z Profile: FULL_PROFILE 2022-11-23T01:34:37.2200251Z Float Round Mode: NEAR 2022-11-23T01:34:37.2201070Z Max Queue Number: 0(0x0) 2022-11-23T01:34:37.2201869Z Queue Min Size: 0(0x0) 2022-11-23T01:34:37.2202658Z Queue Max Size: 0(0x0) 2022-11-23T01:34:37.2203455Z Queue Type: MULTI 2022-11-23T01:34:37.2204229Z Node: 1 2022-11-23T01:34:37.2204984Z Device Type: CPU 2022-11-23T01:34:37.2205679Z Cache Info: 2022-11-23T01:34:37.2206402Z L1: 32768(0x8000) KB 2022-11-23T01:34:37.2207165Z Chip ID: 0(0x0) 2022-11-23T01:34:37.2207951Z Cacheline Size: 64(0x40) 2022-11-23T01:34:37.2209279Z Max Clock Freq. (MHz): 2200 2022-11-23T01:34:37.2210044Z BDFID: 0 2022-11-23T01:34:37.2210903Z Internal Node ID: 1 2022-11-23T01:34:37.2211694Z Compute Unit: 16 2022-11-23T01:34:37.2212468Z SIMDs per CU: 0 2022-11-23T01:34:37.2213257Z Shader Engines: 0 2022-11-23T01:34:37.2214063Z Shader Arrs. per Eng.: 0 2022-11-23T01:34:37.2214887Z WatchPts on Addr. Ranges:1 2022-11-23T01:34:37.2215621Z Features: None 2022-11-23T01:34:37.2216276Z Pool Info: 2022-11-23T01:34:37.2216921Z Pool 1 2022-11-23T01:34:37.2217713Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:34:37.2218712Z Size: 66027676(0x3ef809c) KB 2022-11-23T01:34:37.2219506Z Allocatable: TRUE 2022-11-23T01:34:37.2220328Z Alloc Granule: 4KB 2022-11-23T01:34:37.2221153Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2221983Z Accessible by all: TRUE 2022-11-23T01:34:37.2222702Z Pool 2 2022-11-23T01:34:37.2223500Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:34:37.2224356Z Size: 66027676(0x3ef809c) KB 2022-11-23T01:34:37.2225151Z Allocatable: TRUE 2022-11-23T01:34:37.2225971Z Alloc Granule: 4KB 2022-11-23T01:34:37.2226789Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2227631Z Accessible by all: TRUE 2022-11-23T01:34:37.2228355Z Pool 3 2022-11-23T01:34:37.2229148Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:37.2229979Z Size: 66027676(0x3ef809c) KB 2022-11-23T01:34:37.2230783Z Allocatable: TRUE 2022-11-23T01:34:37.2231600Z Alloc Granule: 4KB 2022-11-23T01:34:37.2232414Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2233244Z Accessible by all: TRUE 2022-11-23T01:34:37.2233958Z ISA Info: 2022-11-23T01:34:37.2234559Z ******* 2022-11-23T01:34:37.2235159Z Agent 3 2022-11-23T01:34:37.2235756Z ******* 2022-11-23T01:34:37.2236783Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2237841Z Uuid: CPU-XX 2022-11-23T01:34:37.2238932Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2239779Z Vendor Name: CPU 2022-11-23T01:34:37.2240590Z Feature: None specified 2022-11-23T01:34:37.2241430Z Profile: FULL_PROFILE 2022-11-23T01:34:37.2242255Z Float Round Mode: NEAR 2022-11-23T01:34:37.2243071Z Max Queue Number: 0(0x0) 2022-11-23T01:34:37.2243864Z Queue Min Size: 0(0x0) 2022-11-23T01:34:37.2244641Z Queue Max Size: 0(0x0) 2022-11-23T01:34:37.2245442Z Queue Type: MULTI 2022-11-23T01:34:37.2246212Z Node: 2 2022-11-23T01:34:37.2247143Z Device Type: CPU 2022-11-23T01:34:37.2247851Z Cache Info: 2022-11-23T01:34:37.2248575Z L1: 32768(0x8000) KB 2022-11-23T01:34:37.2249344Z Chip ID: 0(0x0) 2022-11-23T01:34:37.2250109Z Cacheline Size: 64(0x40) 2022-11-23T01:34:37.2251214Z Max Clock Freq. (MHz): 2200 2022-11-23T01:34:37.2251994Z BDFID: 0 2022-11-23T01:34:37.2252766Z Internal Node ID: 2 2022-11-23T01:34:37.2253562Z Compute Unit: 16 2022-11-23T01:34:37.2254343Z SIMDs per CU: 0 2022-11-23T01:34:37.2255119Z Shader Engines: 0 2022-11-23T01:34:37.2256102Z Shader Arrs. per Eng.: 0 2022-11-23T01:34:37.2256943Z WatchPts on Addr. Ranges:1 2022-11-23T01:34:37.2257466Z Features: None 2022-11-23T01:34:37.2257777Z Pool Info: 2022-11-23T01:34:37.2258075Z Pool 1 2022-11-23T01:34:37.2258367Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:34:37.2258690Z Size: 66052532(0x3efe1b4) KB 2022-11-23T01:34:37.2259001Z Allocatable: TRUE 2022-11-23T01:34:37.2259310Z Alloc Granule: 4KB 2022-11-23T01:34:37.2259618Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2259931Z Accessible by all: TRUE 2022-11-23T01:34:37.2260194Z Pool 2 2022-11-23T01:34:37.2260504Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:34:37.2260833Z Size: 66052532(0x3efe1b4) KB 2022-11-23T01:34:37.2261140Z Allocatable: TRUE 2022-11-23T01:34:37.2261451Z Alloc Granule: 4KB 2022-11-23T01:34:37.2261762Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2262081Z Accessible by all: TRUE 2022-11-23T01:34:37.2262347Z Pool 3 2022-11-23T01:34:37.2262648Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:37.2262972Z Size: 66052532(0x3efe1b4) KB 2022-11-23T01:34:37.2263276Z Allocatable: TRUE 2022-11-23T01:34:37.2263590Z Alloc Granule: 4KB 2022-11-23T01:34:37.2263909Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2264215Z Accessible by all: TRUE 2022-11-23T01:34:37.2264487Z ISA Info: 2022-11-23T01:34:37.2264723Z ******* 2022-11-23T01:34:37.2264950Z Agent 4 2022-11-23T01:34:37.2265177Z ******* 2022-11-23T01:34:37.2265557Z Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2265955Z Uuid: CPU-XX 2022-11-23T01:34:37.2266366Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2266697Z Vendor Name: CPU 2022-11-23T01:34:37.2267009Z Feature: None specified 2022-11-23T01:34:37.2267329Z Profile: FULL_PROFILE 2022-11-23T01:34:37.2267645Z Float Round Mode: NEAR 2022-11-23T01:34:37.2268040Z Max Queue Number: 0(0x0) 2022-11-23T01:34:37.2268348Z Queue Min Size: 0(0x0) 2022-11-23T01:34:37.2268648Z Queue Max Size: 0(0x0) 2022-11-23T01:34:37.2268952Z Queue Type: MULTI 2022-11-23T01:34:37.2269244Z Node: 3 2022-11-23T01:34:37.2269539Z Device Type: CPU 2022-11-23T01:34:37.2269797Z Cache Info: 2022-11-23T01:34:37.2270071Z L1: 32768(0x8000) KB 2022-11-23T01:34:37.2270360Z Chip ID: 0(0x0) 2022-11-23T01:34:37.2270661Z Cacheline Size: 64(0x40) 2022-11-23T01:34:37.2270963Z Max Clock Freq. (MHz): 2200 2022-11-23T01:34:37.2271299Z BDFID: 0 2022-11-23T01:34:37.2271588Z Internal Node ID: 3 2022-11-23T01:34:37.2271893Z Compute Unit: 16 2022-11-23T01:34:37.2272189Z SIMDs per CU: 0 2022-11-23T01:34:37.2272487Z Shader Engines: 0 2022-11-23T01:34:37.2272799Z Shader Arrs. per Eng.: 0 2022-11-23T01:34:37.2273115Z WatchPts on Addr. Ranges:1 2022-11-23T01:34:37.2273397Z Features: None 2022-11-23T01:34:37.2273638Z Pool Info: 2022-11-23T01:34:37.2273883Z Pool 1 2022-11-23T01:34:37.2274183Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:34:37.2274508Z Size: 66051008(0x3efdbc0) KB 2022-11-23T01:34:37.2274815Z Allocatable: TRUE 2022-11-23T01:34:37.2275133Z Alloc Granule: 4KB 2022-11-23T01:34:37.2275436Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2275757Z Accessible by all: TRUE 2022-11-23T01:34:37.2276033Z Pool 2 2022-11-23T01:34:37.2276340Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2022-11-23T01:34:37.2276669Z Size: 66051008(0x3efdbc0) KB 2022-11-23T01:34:37.2276979Z Allocatable: TRUE 2022-11-23T01:34:37.2277282Z Alloc Granule: 4KB 2022-11-23T01:34:37.2277592Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2277908Z Accessible by all: TRUE 2022-11-23T01:34:37.2278183Z Pool 3 2022-11-23T01:34:37.2278497Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:37.2278823Z Size: 66051008(0x3efdbc0) KB 2022-11-23T01:34:37.2279122Z Allocatable: TRUE 2022-11-23T01:34:37.2279431Z Alloc Granule: 4KB 2022-11-23T01:34:37.2279742Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2280058Z Accessible by all: TRUE 2022-11-23T01:34:37.2280334Z ISA Info: 2022-11-23T01:34:37.2280569Z ******* 2022-11-23T01:34:37.2280790Z Agent 5 2022-11-23T01:34:37.2281020Z ******* 2022-11-23T01:34:37.2281282Z Name: gfx906 2022-11-23T01:34:37.2281681Z Uuid: GPU-974210a172e62127 2022-11-23T01:34:37.2281989Z Marketing Name: 2022-11-23T01:34:37.2282350Z Vendor Name: AMD 2022-11-23T01:34:37.2282657Z Feature: KERNEL_DISPATCH 2022-11-23T01:34:37.2282983Z Profile: BASE_PROFILE 2022-11-23T01:34:37.2283303Z Float Round Mode: NEAR 2022-11-23T01:34:37.2283613Z Max Queue Number: 128(0x80) 2022-11-23T01:34:37.2283918Z Queue Min Size: 64(0x40) 2022-11-23T01:34:37.2284220Z Queue Max Size: 131072(0x20000) 2022-11-23T01:34:37.2284527Z Queue Type: MULTI 2022-11-23T01:34:37.2284815Z Node: 4 2022-11-23T01:34:37.2285108Z Device Type: GPU 2022-11-23T01:34:37.2285379Z Cache Info: 2022-11-23T01:34:37.2285704Z L1: 16(0x10) KB 2022-11-23T01:34:37.2285996Z Chip ID: 26273(0x66a1) 2022-11-23T01:34:37.2286299Z Cacheline Size: 64(0x40) 2022-11-23T01:34:37.2286594Z Max Clock Freq. (MHz): 1725 2022-11-23T01:34:37.2286884Z BDFID: 8960 2022-11-23T01:34:37.2287179Z Internal Node ID: 4 2022-11-23T01:34:37.2287475Z Compute Unit: 60 2022-11-23T01:34:37.2287777Z SIMDs per CU: 4 2022-11-23T01:34:37.2288077Z Shader Engines: 4 2022-11-23T01:34:37.2288378Z Shader Arrs. per Eng.: 1 2022-11-23T01:34:37.2288697Z WatchPts on Addr. Ranges:4 2022-11-23T01:34:37.2289003Z Features: KERNEL_DISPATCH 2022-11-23T01:34:37.2289313Z Fast F16 Operation: TRUE 2022-11-23T01:34:37.2289626Z Wavefront Size: 64(0x40) 2022-11-23T01:34:37.2289938Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:37.2290229Z Workgroup Max Size per Dimension: 2022-11-23T01:34:37.2290583Z x 1024(0x400) 2022-11-23T01:34:37.2290871Z y 1024(0x400) 2022-11-23T01:34:37.2291148Z z 1024(0x400) 2022-11-23T01:34:37.2291445Z Max Waves Per CU: 40(0x28) 2022-11-23T01:34:37.2291858Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:34:37.2292179Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:37.2292459Z Grid Max Size per Dimension: 2022-11-23T01:34:37.2292757Z x 4294967295(0xffffffff) 2022-11-23T01:34:37.2293056Z y 4294967295(0xffffffff) 2022-11-23T01:34:37.2293352Z z 4294967295(0xffffffff) 2022-11-23T01:34:37.2293664Z Max fbarriers/Workgrp: 32 2022-11-23T01:34:37.2293937Z Pool Info: 2022-11-23T01:34:37.2294175Z Pool 1 2022-11-23T01:34:37.2294476Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:37.2294801Z Size: 33538048(0x1ffc000) KB 2022-11-23T01:34:37.2295108Z Allocatable: TRUE 2022-11-23T01:34:37.2295419Z Alloc Granule: 4KB 2022-11-23T01:34:37.2295729Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2296039Z Accessible by all: FALSE 2022-11-23T01:34:37.2296500Z Pool 2 2022-11-23T01:34:37.2296805Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:34:37.2297126Z Size: 33538048(0x1ffc000) KB 2022-11-23T01:34:37.2297432Z Allocatable: TRUE 2022-11-23T01:34:37.2297743Z Alloc Granule: 4KB 2022-11-23T01:34:37.2298046Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2298365Z Accessible by all: FALSE 2022-11-23T01:34:37.2298642Z Pool 3 2022-11-23T01:34:37.2298920Z Segment: GROUP 2022-11-23T01:34:37.2299219Z Size: 64(0x40) KB 2022-11-23T01:34:37.2299525Z Allocatable: FALSE 2022-11-23T01:34:37.2299879Z Alloc Granule: 0KB 2022-11-23T01:34:37.2300189Z Alloc Alignment: 0KB 2022-11-23T01:34:37.2300505Z Accessible by all: FALSE 2022-11-23T01:34:37.2300783Z ISA Info: 2022-11-23T01:34:37.2301026Z ISA 1 2022-11-23T01:34:37.2301455Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:34:37.2301827Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:34:37.2302162Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:34:37.2302494Z Default Rounding Mode: NEAR 2022-11-23T01:34:37.2302823Z Default Rounding Mode: NEAR 2022-11-23T01:34:37.2303138Z Fast f16: TRUE 2022-11-23T01:34:37.2303458Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:37.2303760Z Workgroup Max Size per Dimension: 2022-11-23T01:34:37.2304057Z x 1024(0x400) 2022-11-23T01:34:37.2304344Z y 1024(0x400) 2022-11-23T01:34:37.2304632Z z 1024(0x400) 2022-11-23T01:34:37.2304938Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:37.2305234Z Grid Max Size per Dimension: 2022-11-23T01:34:37.2305535Z x 4294967295(0xffffffff) 2022-11-23T01:34:37.2305830Z y 4294967295(0xffffffff) 2022-11-23T01:34:37.2306135Z z 4294967295(0xffffffff) 2022-11-23T01:34:37.2306445Z FBarrier Max Size: 32 2022-11-23T01:34:37.2306705Z ******* 2022-11-23T01:34:37.2306942Z Agent 6 2022-11-23T01:34:37.2307170Z ******* 2022-11-23T01:34:37.2307428Z Name: gfx906 2022-11-23T01:34:37.2307823Z Uuid: GPU-eb9c20c172e620f6 2022-11-23T01:34:37.2308139Z Marketing Name: 2022-11-23T01:34:37.2308444Z Vendor Name: AMD 2022-11-23T01:34:37.2308756Z Feature: KERNEL_DISPATCH 2022-11-23T01:34:37.2309076Z Profile: BASE_PROFILE 2022-11-23T01:34:37.2309386Z Float Round Mode: NEAR 2022-11-23T01:34:37.2309697Z Max Queue Number: 128(0x80) 2022-11-23T01:34:37.2309998Z Queue Min Size: 64(0x40) 2022-11-23T01:34:37.2310301Z Queue Max Size: 131072(0x20000) 2022-11-23T01:34:37.2310667Z Queue Type: MULTI 2022-11-23T01:34:37.2310964Z Node: 5 2022-11-23T01:34:37.2311255Z Device Type: GPU 2022-11-23T01:34:37.2311518Z Cache Info: 2022-11-23T01:34:37.2311794Z L1: 16(0x10) KB 2022-11-23T01:34:37.2312086Z Chip ID: 26273(0x66a1) 2022-11-23T01:34:37.2312382Z Cacheline Size: 64(0x40) 2022-11-23T01:34:37.2312687Z Max Clock Freq. (MHz): 1725 2022-11-23T01:34:37.2312981Z BDFID: 25344 2022-11-23T01:34:37.2313267Z Internal Node ID: 5 2022-11-23T01:34:37.2313568Z Compute Unit: 60 2022-11-23T01:34:37.2313907Z SIMDs per CU: 4 2022-11-23T01:34:37.2314214Z Shader Engines: 4 2022-11-23T01:34:37.2314519Z Shader Arrs. per Eng.: 1 2022-11-23T01:34:37.2314837Z WatchPts on Addr. Ranges:4 2022-11-23T01:34:37.2315130Z Features: KERNEL_DISPATCH 2022-11-23T01:34:37.2315445Z Fast F16 Operation: TRUE 2022-11-23T01:34:37.2315755Z Wavefront Size: 64(0x40) 2022-11-23T01:34:37.2316065Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:37.2316364Z Workgroup Max Size per Dimension: 2022-11-23T01:34:37.2316661Z x 1024(0x400) 2022-11-23T01:34:37.2316933Z y 1024(0x400) 2022-11-23T01:34:37.2317216Z z 1024(0x400) 2022-11-23T01:34:37.2317521Z Max Waves Per CU: 40(0x28) 2022-11-23T01:34:37.2317933Z Max Work-item Per CU: 2560(0xa00) 2022-11-23T01:34:37.2318254Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:37.2318545Z Grid Max Size per Dimension: 2022-11-23T01:34:37.2318839Z x 4294967295(0xffffffff) 2022-11-23T01:34:37.2319130Z y 4294967295(0xffffffff) 2022-11-23T01:34:37.2319428Z z 4294967295(0xffffffff) 2022-11-23T01:34:37.2319738Z Max fbarriers/Workgrp: 32 2022-11-23T01:34:37.2320016Z Pool Info: 2022-11-23T01:34:37.2320263Z Pool 1 2022-11-23T01:34:37.2320568Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2022-11-23T01:34:37.2320882Z Size: 33538048(0x1ffc000) KB 2022-11-23T01:34:37.2321201Z Allocatable: TRUE 2022-11-23T01:34:37.2321513Z Alloc Granule: 4KB 2022-11-23T01:34:37.2321827Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2322147Z Accessible by all: FALSE 2022-11-23T01:34:37.2322422Z Pool 2 2022-11-23T01:34:37.2322716Z Segment: GLOBAL; FLAGS: FINE GRAINED 2022-11-23T01:34:37.2323036Z Size: 33538048(0x1ffc000) KB 2022-11-23T01:34:37.2323346Z Allocatable: TRUE 2022-11-23T01:34:37.2323662Z Alloc Granule: 4KB 2022-11-23T01:34:37.2323973Z Alloc Alignment: 4KB 2022-11-23T01:34:37.2324289Z Accessible by all: FALSE 2022-11-23T01:34:37.2324557Z Pool 3 2022-11-23T01:34:37.2324896Z Segment: GROUP 2022-11-23T01:34:37.2325203Z Size: 64(0x40) KB 2022-11-23T01:34:37.2325509Z Allocatable: FALSE 2022-11-23T01:34:37.2325818Z Alloc Granule: 0KB 2022-11-23T01:34:37.2326132Z Alloc Alignment: 0KB 2022-11-23T01:34:37.2326456Z Accessible by all: FALSE 2022-11-23T01:34:37.2326723Z ISA Info: 2022-11-23T01:34:37.2326968Z ISA 1 2022-11-23T01:34:37.2327391Z Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- 2022-11-23T01:34:37.2327764Z Machine Models: HSA_MACHINE_MODEL_LARGE 2022-11-23T01:34:37.2328109Z Profiles: HSA_PROFILE_BASE 2022-11-23T01:34:37.2328492Z Default Rounding Mode: NEAR 2022-11-23T01:34:37.2328814Z Default Rounding Mode: NEAR 2022-11-23T01:34:37.2329126Z Fast f16: TRUE 2022-11-23T01:34:37.2329434Z Workgroup Max Size: 1024(0x400) 2022-11-23T01:34:37.2329738Z Workgroup Max Size per Dimension: 2022-11-23T01:34:37.2330037Z x 1024(0x400) 2022-11-23T01:34:37.2330329Z y 1024(0x400) 2022-11-23T01:34:37.2330666Z z 1024(0x400) 2022-11-23T01:34:37.2330975Z Grid Max Size: 4294967295(0xffffffff) 2022-11-23T01:34:37.2331270Z Grid Max Size per Dimension: 2022-11-23T01:34:37.2331568Z x 4294967295(0xffffffff) 2022-11-23T01:34:37.2331883Z y 4294967295(0xffffffff) 2022-11-23T01:34:37.2332192Z z 4294967295(0xffffffff) 2022-11-23T01:34:37.2332541Z FBarrier Max Size: 32 2022-11-23T01:34:37.2332801Z *** Done *** 2022-11-23T01:34:37.2333048Z + rocminfo 2022-11-23T01:34:37.2333363Z + grep -E 'Name:.*\sgfx|Marketing' 2022-11-23T01:34:37.2924969Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2929564Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2931993Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2934044Z Marketing Name: AMD EPYC 7601 32-Core Processor 2022-11-23T01:34:37.2935542Z Name: gfx906 2022-11-23T01:34:37.2937026Z Marketing Name: 2022-11-23T01:34:37.2938876Z Name: gfx906 2022-11-23T01:34:37.2940214Z Marketing Name: 2022-11-23T01:34:37.3067425Z + [[ linux-focal-rocm5.2-py3.8 != *-bazel-* ]] 2022-11-23T01:34:37.3067956Z + pip_install --user ninja==1.10.2 2022-11-23T01:34:37.3068569Z + pip install --progress-bar off --user ninja==1.10.2 2022-11-23T01:34:37.8354981Z Collecting ninja==1.10.2 2022-11-23T01:34:37.8844443Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2022-11-23T01:34:38.7431775Z Installing collected packages: ninja 2022-11-23T01:34:38.7532975Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2022-11-23T01:34:38.7534837Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-11-23T01:34:38.7575886Z Successfully installed ninja-1.10.2 2022-11-23T01:34:38.8734667Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:34:38.8736758Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-11-23T01:34:38.8738725Z + [[ linux-focal-rocm5.2-py3.8 == *asan* ]] 2022-11-23T01:34:38.8739925Z + [[ linux-focal-rocm5.2-py3.8 == *-tsan* ]] 2022-11-23T01:34:38.8740933Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2022-11-23T01:34:38.8741831Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2022-11-23T01:34:38.8758313Z + [[ linux-focal-rocm5.2-py3.8 == *tbb* ]] 2022-11-23T01:34:38.8795374Z + [[ linux-focal-rocm5.2-py3.8 == *libtorch* ]] 2022-11-23T01:34:38.8796594Z + [[ linux-focal-rocm5.2-py3.8 == *-bazel-* ]] 2022-11-23T01:34:38.8799704Z + [[ linux-focal-rocm5.2-py3.8 == *-tsan* ]] 2022-11-23T01:34:38.8800891Z + cd test 2022-11-23T01:34:38.8802009Z + python -c 'import torch; print(torch.__config__.show())' 2022-11-23T01:34:40.4062579Z PyTorch built with: 2022-11-23T01:34:40.4063895Z - GCC 9.4 2022-11-23T01:34:40.4064955Z - C++ Version: 201402 2022-11-23T01:34:40.4066589Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-11-23T01:34:40.4068158Z - Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289) 2022-11-23T01:34:40.4069226Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-11-23T01:34:40.4070203Z - LAPACK is enabled (usually provided by MKL) 2022-11-23T01:34:40.4071029Z - NNPACK is enabled 2022-11-23T01:34:40.4071788Z - CPU capability usage: AVX2 2022-11-23T01:34:40.4072541Z - HIP Runtime 5.2.21151 2022-11-23T01:34:40.4073285Z - MIOpen 2.17.0 2022-11-23T01:34:40.4073933Z - Magma 2.6.1 2022-11-23T01:34:40.4081276Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=1.14.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, 2022-11-23T01:34:40.4086403Z 2022-11-23T01:34:42.5810332Z + cd test 2022-11-23T01:34:42.5813070Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2022-11-23T01:34:44.0351115Z ATen/Parallel: 2022-11-23T01:34:44.0385426Z at::get_num_threads() : 32 2022-11-23T01:34:44.0386338Z at::get_num_interop_threads() : 32 2022-11-23T01:34:44.0387119Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-11-23T01:34:44.0388261Z omp_get_max_threads() : 32 2022-11-23T01:34:44.0390094Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-11-23T01:34:44.0391248Z mkl_get_max_threads() : 32 2022-11-23T01:34:44.0392474Z Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289) 2022-11-23T01:34:44.0393522Z std::thread::hardware_concurrency() : 64 2022-11-23T01:34:44.0394316Z Environment variables: 2022-11-23T01:34:44.0395579Z OMP_NUM_THREADS : [not set] 2022-11-23T01:34:44.0396394Z MKL_NUM_THREADS : [not set] 2022-11-23T01:34:44.0397214Z ATen parallel backend: OpenMP 2022-11-23T01:34:44.0397773Z 2022-11-23T01:34:46.2159990Z + [[ distributed == *backward* ]] 2022-11-23T01:34:46.2161285Z + [[ distributed == *xla* ]] 2022-11-23T01:34:46.2162396Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2022-11-23T01:34:46.2164034Z + [[ linux-focal-rocm5.2-py3.8 == *libtorch* ]] 2022-11-23T01:34:46.2164943Z + [[ distributed == distributed ]] 2022-11-23T01:34:46.2165684Z + install_filelock 2022-11-23T01:34:46.2221871Z + pip_install filelock 2022-11-23T01:34:46.2223133Z + pip install --progress-bar off filelock 2022-11-23T01:34:46.7621633Z Collecting filelock 2022-11-23T01:34:46.8049726Z Downloading filelock-3.8.0-py3-none-any.whl (10 kB) 2022-11-23T01:34:47.6704114Z Installing collected packages: filelock 2022-11-23T01:34:47.7034119Z Successfully installed filelock-3.8.0 2022-11-23T01:34:47.8120560Z + install_triton 2022-11-23T01:34:47.8121445Z + local commit 2022-11-23T01:34:47.8122226Z + [[ distributed == *rocm* ]] 2022-11-23T01:34:47.8126682Z ++ get_pinned_commit triton 2022-11-23T01:34:47.8127619Z ++ cat .github/ci_commit_pins/triton.txt 2022-11-23T01:34:47.8159292Z + commit=0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:34:47.8161351Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:34:47.8163348Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:34:48.2650904Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python 2022-11-23T01:34:48.2653944Z Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-3dg6qew_ 2022-11-23T01:34:48.2701772Z Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-3dg6qew_ 2022-11-23T01:34:50.2958189Z Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331' 2022-11-23T01:34:50.3033877Z Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:34:51.0047720Z Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:34:51.5761626Z Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331 2022-11-23T01:34:51.5764001Z Running command git submodule update --init --recursive -q 2022-11-23T01:34:52.7607586Z Preparing metadata (setup.py) ... [?25l- done 2022-11-23T01:34:52.9983901Z [?25hCollecting cmake 2022-11-23T01:34:53.0417708Z Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB) 2022-11-23T01:34:53.5693185Z Requirement already satisfied: filelock in /opt/conda/lib/python3.8/site-packages (from triton==2.0.0) (3.8.0) 2022-11-23T01:34:53.5695998Z Requirement already satisfied: torch in /opt/conda/lib/python3.8/site-packages (from triton==2.0.0) (1.14.0a0+git1cfd385) 2022-11-23T01:34:53.5952847Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (4.4.0) 2022-11-23T01:34:53.5955264Z Requirement already satisfied: networkx in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (2.6.3) 2022-11-23T01:34:53.5957733Z Requirement already satisfied: sympy in /opt/conda/lib/python3.8/site-packages (from torch->triton==2.0.0) (1.11.1) 2022-11-23T01:34:53.6153321Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy->torch->triton==2.0.0) (1.2.1) 2022-11-23T01:34:53.6234161Z Building wheels for collected packages: triton 2022-11-23T01:36:08.5142844Z Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | / done 2022-11-23T01:36:08.5361796Z [?25h Created wheel for triton: filename=triton-2.0.0-cp38-cp38-linux_x86_64.whl size=15414539 sha256=712c8d8f423dd60049b2eaea481534d0fab755d31a71ba4bbcf64a36a9b317f1 2022-11-23T01:36:08.5363943Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/c0/c0/56/bdb2859a55c7764d4e97889d26a8a05b683ef97fe9b1aa7dec 2022-11-23T01:36:08.5389263Z Successfully built triton 2022-11-23T01:36:09.4191602Z Installing collected packages: cmake, triton 2022-11-23T01:36:10.6320129Z Successfully installed cmake-3.25.0 triton-2.0.0 2022-11-23T01:36:10.7521614Z + pip_install --user jinja2 2022-11-23T01:36:10.7522694Z + pip install --progress-bar off --user jinja2 2022-11-23T01:36:11.2924771Z Collecting jinja2 2022-11-23T01:36:11.3372428Z Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) 2022-11-23T01:36:11.3603057Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.8/site-packages (from jinja2) (2.1.1) 2022-11-23T01:36:12.2003562Z Installing collected packages: jinja2 2022-11-23T01:36:12.2840270Z Successfully installed jinja2-3.1.2 2022-11-23T01:36:12.3967758Z + test_distributed 2022-11-23T01:36:12.3969135Z + echo 'Testing distributed python tests' 2022-11-23T01:36:12.3979222Z Testing distributed python tests 2022-11-23T01:36:12.3980771Z + python test/run_test.py --distributed-tests --shard 2 2 --verbose 2022-11-23T01:36:15.1166724Z Ignoring disabled issues: [] 2022-11-23T01:36:15.1374055Z Excluding distributed/rpc/test_faulty_agent on ROCm 2022-11-23T01:36:15.1376628Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm 2022-11-23T01:36:15.1377684Z Excluding distributed/rpc/test_share_memory on ROCm 2022-11-23T01:36:15.1378752Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm 2022-11-23T01:36:15.1379853Z Excluding distributed/_shard/sharding_plan/test_sharding_plan on ROCm 2022-11-23T01:36:15.1381017Z Excluding distributed/_shard/sharded_tensor/test_megatron_prototype on ROCm 2022-11-23T01:36:15.1382176Z Excluding distributed/_shard/sharded_tensor/test_sharded_tensor on ROCm 2022-11-23T01:36:15.1383393Z Excluding distributed/_shard/sharded_tensor/test_sharded_tensor_reshard on ROCm 2022-11-23T01:36:15.1384539Z Excluding distributed/_shard/sharded_tensor/ops/test_chunk on ROCm 2022-11-23T01:36:15.1385755Z Excluding distributed/_shard/sharded_tensor/ops/test_elementwise_ops on ROCm 2022-11-23T01:36:15.1387019Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding on ROCm 2022-11-23T01:36:15.1388153Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding_bag on ROCm 2022-11-23T01:36:15.1389302Z Excluding distributed/_shard/sharded_tensor/ops/test_binary_cmp on ROCm 2022-11-23T01:36:15.1390413Z Excluding distributed/_shard/sharded_tensor/ops/test_init on ROCm 2022-11-23T01:36:15.1391832Z Excluding distributed/_shard/sharded_tensor/ops/test_linear on ROCm 2022-11-23T01:36:15.1392925Z Excluding distributed/_shard/sharded_tensor/ops/test_math_ops on ROCm 2022-11-23T01:36:15.1394045Z Excluding distributed/_shard/sharded_tensor/ops/test_matrix_ops on ROCm 2022-11-23T01:36:15.1395193Z Excluding distributed/_shard/sharded_tensor/ops/test_softmax on ROCm 2022-11-23T01:36:15.1396653Z Excluding distributed/_shard/sharded_optim/test_sharded_optim on ROCm 2022-11-23T01:36:15.1398125Z Excluding distributed/_shard/test_partial_tensor on ROCm 2022-11-23T01:36:15.1399388Z Excluding distributed/_shard/test_replicated_tensor on ROCm 2022-11-23T01:36:15.1457537Z ##[warning] Gathered no stats from artifacts. Proceeding with default sharding plan. 2022-11-23T01:36:15.1459531Z Selected tests: 2022-11-23T01:36:15.1460363Z distributed/algorithms/quantization/test_quantization 2022-11-23T01:36:15.1461225Z distributed/test_distributed_spawn 2022-11-23T01:36:15.1461967Z distributed/test_store 2022-11-23T01:36:15.1462633Z distributed/test_nccl 2022-11-23T01:36:15.1463313Z distributed/test_launcher 2022-11-23T01:36:15.1464038Z distributed/test_data_parallel 2022-11-23T01:36:15.1464766Z distributed/test_c10d_spawn_nccl 2022-11-23T01:36:15.1465496Z distributed/test_c10d_pypg 2022-11-23T01:36:15.1466232Z distributed/test_c10d_nccl 2022-11-23T01:36:15.1467529Z distributed/test_c10d_error_logger 2022-11-23T01:36:15.1468331Z distributed/pipeline/sync/test_worker 2022-11-23T01:36:15.1469144Z distributed/pipeline/sync/test_stream 2022-11-23T01:36:15.1469928Z distributed/pipeline/sync/test_pipe 2022-11-23T01:36:15.1470890Z distributed/pipeline/sync/test_microbatch 2022-11-23T01:36:15.1471742Z distributed/pipeline/sync/test_dependency 2022-11-23T01:36:15.1472558Z distributed/pipeline/sync/test_copy 2022-11-23T01:36:15.1473353Z distributed/pipeline/sync/test_bugs 2022-11-23T01:36:15.1474242Z distributed/pipeline/sync/skip/test_verify_skippables 2022-11-23T01:36:15.1475167Z distributed/pipeline/sync/skip/test_stash_pop 2022-11-23T01:36:15.1476003Z distributed/pipeline/sync/skip/test_leak 2022-11-23T01:36:15.1476853Z distributed/pipeline/sync/skip/test_gpipe 2022-11-23T01:36:15.1477739Z distributed/optim/test_zero_redundancy_optimizer 2022-11-23T01:36:15.1478577Z distributed/nn/jit/test_instantiator 2022-11-23T01:36:15.1479545Z distributed/fsdp/test_utils 2022-11-23T01:36:15.1480348Z distributed/fsdp/test_fsdp_use_orig_params 2022-11-23T01:36:15.1481142Z distributed/fsdp/test_fsdp_traversal 2022-11-23T01:36:15.1481985Z distributed/fsdp/test_fsdp_summon_full_params 2022-11-23T01:36:15.1482864Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2022-11-23T01:36:15.1483678Z distributed/fsdp/test_fsdp_overlap 2022-11-23T01:36:15.1484513Z distributed/fsdp/test_fsdp_multiple_wrapping 2022-11-23T01:36:15.1485349Z distributed/fsdp/test_fsdp_mixed_precision 2022-11-23T01:36:15.1486147Z distributed/fsdp/test_fsdp_meta 2022-11-23T01:36:15.1486900Z distributed/fsdp/test_fsdp_input 2022-11-23T01:36:15.1487665Z distributed/fsdp/test_fsdp_grad_acc 2022-11-23T01:36:15.1488497Z distributed/fsdp/test_fsdp_freezing_weights 2022-11-23T01:36:15.1489321Z distributed/fsdp/test_fsdp_exec_order 2022-11-23T01:36:15.1490086Z distributed/fsdp/test_fsdp_comm_hooks 2022-11-23T01:36:15.1491307Z distributed/fsdp/test_fsdp_clip_grad_norm 2022-11-23T01:36:15.1492112Z distributed/fsdp/test_fsdp_apply 2022-11-23T01:36:15.1492910Z distributed/fsdp/test_checkpoint_wrapper 2022-11-23T01:36:15.1493730Z distributed/elastic/utils/logging_test 2022-11-23T01:36:15.1494554Z distributed/elastic/timer/local_timer_test 2022-11-23T01:36:15.1495331Z distributed/elastic/timer/api_test 2022-11-23T01:36:15.1496111Z distributed/elastic/metrics/api_test 2022-11-23T01:36:15.1496897Z distributed/checkpoint/test_utils 2022-11-23T01:36:15.1497677Z distributed/checkpoint/test_planner 2022-11-23T01:36:15.1498560Z distributed/checkpoint/test_file_system_checkpoint 2022-11-23T01:36:15.1499395Z distributed/checkpoint/test_checkpoint 2022-11-23T01:36:15.1500280Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2022-11-23T01:36:15.1501143Z distributed/_tensor/test_tp_sharding_ops 2022-11-23T01:36:15.1501939Z distributed/_tensor/test_redistribute 2022-11-23T01:36:15.1502731Z distributed/_tensor/test_matrix_ops 2022-11-23T01:36:15.1503507Z distributed/_tensor/test_dtensor 2022-11-23T01:36:15.1504247Z distributed/_tensor/test_common_rules 2022-11-23T01:36:15.1505144Z distributed/_tensor/parallel/test_view_sharding_dim_change 2022-11-23T01:36:15.1506035Z distributed/_tensor/parallel/test_tp_examples 2022-11-23T01:36:15.1506681Z distributed/_tensor/parallel/test_2d_parallel 2022-11-23T01:36:15.1507343Z distributed/_shard/sharding_spec/test_sharding_spec 2022-11-23T01:36:15.1507965Z distributed/_composable/test_replicate 2022-11-23T01:36:15.1508522Z distributed/_composable/test_contract 2022-11-23T01:36:15.1548769Z Prioritized test from test file changes. 2022-11-23T01:36:15.1549381Z reordering tests for PR: 2022-11-23T01:36:15.1549890Z prioritized: [] 2022-11-23T01:36:15.1561315Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/test_store', 'distributed/test_nccl', 'distributed/test_launcher', 'distributed/test_data_parallel', 'distributed/test_c10d_spawn_nccl', 'distributed/test_c10d_pypg', 'distributed/test_c10d_nccl', 'distributed/test_c10d_error_logger', 'distributed/pipeline/sync/test_worker', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/nn/jit/test_instantiator', 'distributed/fsdp/test_utils', 'distributed/fsdp/test_fsdp_use_orig_params', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/fsdp/test_fsdp_overlap', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_input', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/fsdp/test_fsdp_apply', 'distributed/fsdp/test_checkpoint_wrapper', 'distributed/elastic/utils/logging_test', 'distributed/elastic/timer/local_timer_test', 'distributed/elastic/timer/api_test', 'distributed/elastic/metrics/api_test', 'distributed/checkpoint/test_utils', 'distributed/checkpoint/test_planner', 'distributed/checkpoint/test_file_system_checkpoint', 'distributed/checkpoint/test_checkpoint', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/_tensor/test_tp_sharding_ops', 'distributed/_tensor/test_redistribute', 'distributed/_tensor/test_matrix_ops', 'distributed/_tensor/test_dtensor', 'distributed/_tensor/test_common_rules', 'distributed/_tensor/parallel/test_view_sharding_dim_change', 'distributed/_tensor/parallel/test_tp_examples', 'distributed/_tensor/parallel/test_2d_parallel', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/_composable/test_replicate', 'distributed/_composable/test_contract'] 2022-11-23T01:36:15.1567985Z 2022-11-23T01:36:15.1569030Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2022-11-23T01:36:15.3386150Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2022-11-23T01:36:15.5407366Z parallel (file granularity) tests: 2022-11-23T01:36:15.5408062Z 2022-11-23T01:36:15.5408636Z serial (file granularity) tests: 2022-11-23T01:36:15.5409528Z distributed/algorithms/quantization/test_quantization 2022-11-23T01:36:15.5410431Z distributed/test_distributed_spawn 2022-11-23T01:36:15.5411326Z distributed/test_store 2022-11-23T01:36:15.5412020Z distributed/test_nccl 2022-11-23T01:36:15.5412681Z distributed/test_launcher 2022-11-23T01:36:15.5413408Z distributed/test_data_parallel 2022-11-23T01:36:15.5414149Z distributed/test_c10d_spawn_nccl 2022-11-23T01:36:15.5414867Z distributed/test_c10d_pypg 2022-11-23T01:36:15.5415559Z distributed/test_c10d_nccl 2022-11-23T01:36:15.5416282Z distributed/test_c10d_error_logger 2022-11-23T01:36:15.5417052Z distributed/pipeline/sync/test_worker 2022-11-23T01:36:15.5417866Z distributed/pipeline/sync/test_stream 2022-11-23T01:36:15.5418664Z distributed/pipeline/sync/test_pipe 2022-11-23T01:36:15.5419495Z distributed/pipeline/sync/test_microbatch 2022-11-23T01:36:15.5420342Z distributed/pipeline/sync/test_dependency 2022-11-23T01:36:15.5421157Z distributed/pipeline/sync/test_copy 2022-11-23T01:36:15.5422000Z distributed/pipeline/sync/test_bugs 2022-11-23T01:36:15.5422883Z distributed/pipeline/sync/skip/test_verify_skippables 2022-11-23T01:36:15.5423722Z distributed/pipeline/sync/skip/test_stash_pop 2022-11-23T01:36:15.5424835Z distributed/pipeline/sync/skip/test_leak 2022-11-23T01:36:15.5425558Z distributed/pipeline/sync/skip/test_gpipe 2022-11-23T01:36:15.5426305Z distributed/optim/test_zero_redundancy_optimizer 2022-11-23T01:36:15.5426998Z distributed/nn/jit/test_instantiator 2022-11-23T01:36:15.5427634Z distributed/fsdp/test_utils 2022-11-23T01:36:15.5428302Z distributed/fsdp/test_fsdp_use_orig_params 2022-11-23T01:36:15.5428988Z distributed/fsdp/test_fsdp_traversal 2022-11-23T01:36:15.5429699Z distributed/fsdp/test_fsdp_summon_full_params 2022-11-23T01:36:15.5430446Z distributed/fsdp/test_fsdp_sharded_grad_scaler 2022-11-23T01:36:15.5431122Z distributed/fsdp/test_fsdp_overlap 2022-11-23T01:36:15.5431828Z distributed/fsdp/test_fsdp_multiple_wrapping 2022-11-23T01:36:15.5432558Z distributed/fsdp/test_fsdp_mixed_precision 2022-11-23T01:36:15.5433229Z distributed/fsdp/test_fsdp_meta 2022-11-23T01:36:15.5433870Z distributed/fsdp/test_fsdp_input 2022-11-23T01:36:15.5434515Z distributed/fsdp/test_fsdp_grad_acc 2022-11-23T01:36:15.5435352Z distributed/fsdp/test_fsdp_freezing_weights 2022-11-23T01:36:15.5436047Z distributed/fsdp/test_fsdp_exec_order 2022-11-23T01:36:15.5436720Z distributed/fsdp/test_fsdp_comm_hooks 2022-11-23T01:36:15.5437407Z distributed/fsdp/test_fsdp_clip_grad_norm 2022-11-23T01:36:15.5438075Z distributed/fsdp/test_fsdp_apply 2022-11-23T01:36:15.5438737Z distributed/fsdp/test_checkpoint_wrapper 2022-11-23T01:36:15.5439426Z distributed/elastic/utils/logging_test 2022-11-23T01:36:15.5440122Z distributed/elastic/timer/local_timer_test 2022-11-23T01:36:15.5440792Z distributed/elastic/timer/api_test 2022-11-23T01:36:15.5441451Z distributed/elastic/metrics/api_test 2022-11-23T01:36:15.5442114Z distributed/checkpoint/test_utils 2022-11-23T01:36:15.5442757Z distributed/checkpoint/test_planner 2022-11-23T01:36:15.5443492Z distributed/checkpoint/test_file_system_checkpoint 2022-11-23T01:36:15.5444219Z distributed/checkpoint/test_checkpoint 2022-11-23T01:36:15.5444978Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2022-11-23T01:36:15.5445723Z distributed/_tensor/test_tp_sharding_ops 2022-11-23T01:36:15.5446407Z distributed/_tensor/test_redistribute 2022-11-23T01:36:15.5447052Z distributed/_tensor/test_matrix_ops 2022-11-23T01:36:15.5447693Z distributed/_tensor/test_dtensor 2022-11-23T01:36:15.5448341Z distributed/_tensor/test_common_rules 2022-11-23T01:36:15.5449103Z distributed/_tensor/parallel/test_view_sharding_dim_change 2022-11-23T01:36:15.5449890Z distributed/_tensor/parallel/test_tp_examples 2022-11-23T01:36:15.5450706Z distributed/_tensor/parallel/test_2d_parallel 2022-11-23T01:36:15.5451460Z distributed/_shard/sharding_spec/test_sharding_spec 2022-11-23T01:36:15.5452193Z distributed/_composable/test_replicate 2022-11-23T01:36:15.5452873Z distributed/_composable/test_contract 2022-11-23T01:36:17.8872905Z Ignoring disabled issues: [] 2022-11-23T01:36:20.3230677Z Running distributed/algorithms/quantization/test_quantization ... [2022-11-23 01:36:20.322232] 2022-11-23T01:36:20.3399302Z MPI not available -- MPI backend tests will be skipped 2022-11-23T01:36:20.3401000Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2} 2022-11-23T01:36:20.3402119Z Shard 2: test should be run in 1 2022-11-23T01:36:20.3408957Z Running distributed tests for the nccl backend with env init_method in shard 2 of 2 2022-11-23T01:36:20.3420685Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:36:20.341422] 2022-11-23T01:37:06.7472167Z 2022-11-23T01:37:06.7473497Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:06.7493634Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_54zle3ae) 2022-11-23T01:37:06.7497292Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2022-11-23T01:37:06.7499686Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:06.7500632Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:06.7501559Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:06.7502475Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:06.7503398Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:06.7504391Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:06.7505476Z 2022-11-23T01:37:06.7507343Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:06.7509211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7510398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7511999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7513333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7514159Z 2022-11-23T01:37:06.7514533Z Running tests... 2022-11-23T01:37:06.7515962Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7517689Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s) 2022-11-23T01:37:06.7518673Z 2022-11-23T01:37:06.7519710Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7520814Z Ran 1 test in 0.001s 2022-11-23T01:37:06.7521354Z 2022-11-23T01:37:06.7521700Z OK (skipped=1) 2022-11-23T01:37:06.7522158Z 2022-11-23T01:37:06.7522509Z Generating XML reports... 2022-11-23T01:37:06.7524843Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013623.xml 2022-11-23T01:37:06.7527273Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:06.7529370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7531017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7532942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7534183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7534767Z 2022-11-23T01:37:06.7535015Z Running tests... 2022-11-23T01:37:06.7536091Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7537325Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s) 2022-11-23T01:37:06.7538052Z 2022-11-23T01:37:06.7538755Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7539577Z Ran 1 test in 0.001s 2022-11-23T01:37:06.7539949Z 2022-11-23T01:37:06.7540191Z OK (skipped=1) 2022-11-23T01:37:06.7540556Z 2022-11-23T01:37:06.7540839Z Generating XML reports... 2022-11-23T01:37:06.7542575Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013627.xml 2022-11-23T01:37:06.7544587Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:06.7546625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7547795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7549359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7550564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7551138Z 2022-11-23T01:37:06.7551378Z Running tests... 2022-11-23T01:37:06.7552458Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7553781Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1234 2022-11-23T01:37:06.7555140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1235 2022-11-23T01:37:06.7556455Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:06.7558393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7559553Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7561084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7562295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7563422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:06.7565093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7566247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7567804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7569030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7570151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:06.7571702Z skip: Test skipped for ROCm (4.504s) 2022-11-23T01:37:06.7572172Z 2022-11-23T01:37:06.7572912Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7573745Z Ran 1 test in 4.505s 2022-11-23T01:37:06.7574132Z 2022-11-23T01:37:06.7574374Z OK (skipped=1) 2022-11-23T01:37:06.7574743Z 2022-11-23T01:37:06.7575028Z Generating XML reports... 2022-11-23T01:37:06.7576789Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013631.xml 2022-11-23T01:37:06.7578792Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:06.7580597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7581758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7583306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7584514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7585087Z 2022-11-23T01:37:06.7585326Z Running tests... 2022-11-23T01:37:06.7586400Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7587709Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1435 2022-11-23T01:37:06.7588785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1436 2022-11-23T01:37:06.7589397Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:06.7590144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7590589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7591184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7591642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7592068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:06.7592681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7593116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7593697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7594224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7594654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:06.7595023Z skip: Test skipped for ROCm (4.702s) 2022-11-23T01:37:06.7595197Z 2022-11-23T01:37:06.7595473Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7595773Z Ran 1 test in 4.702s 2022-11-23T01:37:06.7595919Z 2022-11-23T01:37:06.7596014Z OK (skipped=1) 2022-11-23T01:37:06.7596155Z 2022-11-23T01:37:06.7596265Z Generating XML reports... 2022-11-23T01:37:06.7596918Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013640.xml 2022-11-23T01:37:06.7597669Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:06.7598342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7598786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7599364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7599830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7600049Z 2022-11-23T01:37:06.7600143Z Running tests... 2022-11-23T01:37:06.7600554Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7601062Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1636 2022-11-23T01:37:06.7601582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1637 2022-11-23T01:37:06.7602072Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:06.7602739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7603165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7603752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7604207Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7604637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:06.7605121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:37:06.7605754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7606191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7606820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7607287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7607714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:06.7608182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:37:06.7608844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:06.7609363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:37:06.7610025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:06.7610605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:37:06.7611632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:06.7612323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:06.7612900Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T01:37:06.7613401Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T01:37:06.7613767Z ok (5.107s) 2022-11-23T01:37:06.7613931Z 2022-11-23T01:37:06.7614249Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7614619Z Ran 1 test in 5.108s 2022-11-23T01:37:06.7614788Z 2022-11-23T01:37:06.7614879Z OK 2022-11-23T01:37:06.7615028Z 2022-11-23T01:37:06.7615158Z Generating XML reports... 2022-11-23T01:37:06.7615940Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013648.xml 2022-11-23T01:37:06.7616848Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:06.7617654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7618176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7618876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7619415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7619672Z 2022-11-23T01:37:06.7619779Z Running tests... 2022-11-23T01:37:06.7620266Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7620875Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1852 2022-11-23T01:37:06.7621507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1853 2022-11-23T01:37:06.7622100Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:06.7622901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7623427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7624121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7624673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7625187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:06.7625756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:37:06.7626600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:06.7627083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:06.7627666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:06.7628112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:06.7628536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:06.7629012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:37:06.7629673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:06.7630192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:37:06.7630846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:06.7631428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:37:06.7632082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:06.7632760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:06.7633323Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T01:37:06.7633817Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T01:37:06.7634127Z ok (5.100s) 2022-11-23T01:37:06.7634259Z 2022-11-23T01:37:06.7634525Z ---------------------------------------------------------------------- 2022-11-23T01:37:06.7634836Z Ran 1 test in 5.100s 2022-11-23T01:37:06.7634982Z 2022-11-23T01:37:06.7635063Z OK 2022-11-23T01:37:06.7635173Z 2022-11-23T01:37:06.7635286Z Generating XML reports... 2022-11-23T01:37:06.7635934Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013657.xml 2022-11-23T01:37:06.7636309Z 2022-11-23T01:37:06.7636655Z ##[endgroup] 2022-11-23T01:37:06.7637372Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_54zle3ae) 2022-11-23T01:37:06.7637788Z 2022-11-23T01:37:06.7637989Z Running distributed tests for the nccl backend with file init_method in shard 2 of 2 2022-11-23T01:37:06.7638795Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:37:06.748249] 2022-11-23T01:37:53.7152253Z 2022-11-23T01:37:53.7153516Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization 2022-11-23T01:37:53.7156282Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_zjnkx3h1) 2022-11-23T01:37:53.7159329Z , <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]> 2022-11-23T01:37:53.7161696Z test_all_gather_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:53.7162660Z test_all_gather_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:53.7163617Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:53.7165147Z test_all_to_all_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:53.7166139Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:53.7167142Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) 2022-11-23T01:37:53.7168029Z 2022-11-23T01:37:53.7170206Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:53.7173311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7174747Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7176526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7177798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7178399Z 2022-11-23T01:37:53.7178737Z Running tests... 2022-11-23T01:37:53.7180411Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7182054Z test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s) 2022-11-23T01:37:53.7182865Z 2022-11-23T01:37:53.7183851Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7185048Z Ran 1 test in 0.001s 2022-11-23T01:37:53.7185548Z 2022-11-23T01:37:53.7185845Z OK (skipped=1) 2022-11-23T01:37:53.7186330Z 2022-11-23T01:37:53.7186699Z Generating XML reports... 2022-11-23T01:37:53.7188951Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013710.xml 2022-11-23T01:37:53.7191369Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:53.7193259Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7194493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7196098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7197341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7197910Z 2022-11-23T01:37:53.7198342Z Running tests... 2022-11-23T01:37:53.7199274Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7200326Z test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s) 2022-11-23T01:37:53.7200962Z 2022-11-23T01:37:53.7201571Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7202279Z Ran 1 test in 0.001s 2022-11-23T01:37:53.7202620Z 2022-11-23T01:37:53.7202826Z OK (skipped=1) 2022-11-23T01:37:53.7203148Z 2022-11-23T01:37:53.7203389Z Generating XML reports... 2022-11-23T01:37:53.7204919Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013714.xml 2022-11-23T01:37:53.7206682Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:53.7208228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7209228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7210650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7211812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7212413Z 2022-11-23T01:37:53.7212672Z Running tests... 2022-11-23T01:37:53.7213787Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7215374Z test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2266 2022-11-23T01:37:53.7216829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2267 2022-11-23T01:37:53.7218203Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:53.7219907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7220907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7222247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7223298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7224283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:53.7225750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7226865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7228213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7229251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7230222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:53.7231063Z skip: Test skipped for ROCm (4.601s) 2022-11-23T01:37:53.7231459Z 2022-11-23T01:37:53.7232063Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7232772Z Ran 1 test in 4.602s 2022-11-23T01:37:53.7233105Z 2022-11-23T01:37:53.7233322Z OK (skipped=1) 2022-11-23T01:37:53.7233640Z 2022-11-23T01:37:53.7233875Z Generating XML reports... 2022-11-23T01:37:53.7235399Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013718.xml 2022-11-23T01:37:53.7237145Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:53.7238697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7239689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7241018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7242049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7242534Z 2022-11-23T01:37:53.7242752Z Running tests... 2022-11-23T01:37:53.7243668Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7244790Z test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2467 2022-11-23T01:37:53.7245973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2468 2022-11-23T01:37:53.7247091Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:53.7248639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7249633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7251085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7252287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7253462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:53.7255202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7256563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7258187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7259438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7260593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:53.7261599Z skip: Test skipped for ROCm (4.500s) 2022-11-23T01:37:53.7262050Z 2022-11-23T01:37:53.7262800Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7263636Z Ran 1 test in 4.500s 2022-11-23T01:37:53.7264030Z 2022-11-23T01:37:53.7264278Z OK (skipped=1) 2022-11-23T01:37:53.7264652Z 2022-11-23T01:37:53.7264941Z Generating XML reports... 2022-11-23T01:37:53.7266747Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013726.xml 2022-11-23T01:37:53.7269017Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:53.7270905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7272078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7273691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7274962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7275566Z 2022-11-23T01:37:53.7275823Z Running tests... 2022-11-23T01:37:53.7276925Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7278315Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2668 2022-11-23T01:37:53.7279751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2669 2022-11-23T01:37:53.7281075Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:53.7282886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7284088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7285684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7286932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7288088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:53.7289377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:37:53.7291464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7292666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7294308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7295580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7296752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:53.7298078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:37:53.7299897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:53.7301354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:37:53.7303540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:53.7305092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:37:53.7307167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:53.7308591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:53.7309214Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T01:37:53.7309729Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T01:37:53.7310046Z ok (5.104s) 2022-11-23T01:37:53.7310190Z 2022-11-23T01:37:53.7310456Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7310762Z Ran 1 test in 5.104s 2022-11-23T01:37:53.7310999Z 2022-11-23T01:37:53.7311079Z OK 2022-11-23T01:37:53.7311205Z 2022-11-23T01:37:53.7311315Z Generating XML reports... 2022-11-23T01:37:53.7311986Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013734.xml 2022-11-23T01:37:53.7312734Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization 2022-11-23T01:37:53.7313404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7313847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7314432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7314908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7315130Z 2022-11-23T01:37:53.7315228Z Running tests... 2022-11-23T01:37:53.7315675Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7316200Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2884 2022-11-23T01:37:53.7316719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2885 2022-11-23T01:37:53.7317223Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T01:37:53.7317868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7318304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7318892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7319349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7319781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T01:37:53.7320251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T01:37:53.7320883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T01:37:53.7321322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T01:37:53.7321893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T01:37:53.7322349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T01:37:53.7322771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T01:37:53.7323239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T01:37:53.7323957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:53.7324481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T01:37:53.7325135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T01:37:53.7325636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T01:37:53.7326292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:53.7326988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T01:37:53.7327562Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T01:37:53.7328067Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T01:37:53.7328442Z ok (5.486s) 2022-11-23T01:37:53.7328576Z 2022-11-23T01:37:53.7328847Z ---------------------------------------------------------------------- 2022-11-23T01:37:53.7329149Z Ran 1 test in 5.486s 2022-11-23T01:37:53.7329297Z 2022-11-23T01:37:53.7329378Z OK 2022-11-23T01:37:53.7329501Z 2022-11-23T01:37:53.7329611Z Generating XML reports... 2022-11-23T01:37:53.7330261Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123013743.xml 2022-11-23T01:37:53.7330710Z 2022-11-23T01:37:53.7331072Z ##[endgroup] 2022-11-23T01:37:53.7331904Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-quantization-test_quantization_zjnkx3h1) 2022-11-23T01:37:53.7332410Z 2022-11-23T01:37:53.7332549Z Shard 2: gloo should be run in 1 2022-11-23T01:37:53.7333068Z Running distributed/test_distributed_spawn ... [2022-11-23 01:37:53.717523] 2022-11-23T01:37:53.7333641Z MPI not available -- MPI backend tests will be skipped 2022-11-23T01:37:53.7334319Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 1} 2022-11-23T01:37:53.7334784Z Shard 2: test should be run in 1 2022-11-23T01:37:53.7335215Z Running distributed tests for the nccl backend with env init_method in shard 2 of 2 2022-11-23T01:37:53.7336128Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:37:53.730691] 2022-11-23T02:06:51.8670479Z 2022-11-23T02:06:51.8675726Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T02:06:51.8677992Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_dw02y2dq) 2022-11-23T02:06:51.8682914Z 2022-11-23T02:06:51.8794772Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-11-23T02:06:51.8884826Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8886203Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8887288Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8888311Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8889401Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8890714Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8891974Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8893309Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8894757Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8896162Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8897654Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8898960Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8900313Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8901768Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8902971Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8904233Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8905406Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8906516Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8907631Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8909072Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8910364Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8911576Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8912634Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8913655Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8914866Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8916045Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8917091Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8918138Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8919308Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8920544Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8921535Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8922540Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8923678Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8924765Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8925987Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8927142Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8928224Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8929520Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8930695Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8931771Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8932898Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8934179Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8935437Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8936713Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8938024Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8939301Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8940552Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8941762Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8942994Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8944141Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8945367Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8946480Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8947781Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8948827Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8949864Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8950850Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8951785Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8952786Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8953756Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8954716Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8955869Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8956853Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8957718Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8958523Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8959112Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8959725Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8960351Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8960953Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8961477Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8962061Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8962655Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8963245Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8963920Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8964546Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8965132Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8965644Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8966204Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8966784Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8967389Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8967904Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8968407Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8968868Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8969347Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8969847Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8970354Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8971053Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8971571Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8972094Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8972611Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8973125Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8973628Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8974129Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8974649Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8975167Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8975693Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8976217Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8976742Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8977263Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8977753Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8978208Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8978647Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8979064Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8979443Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8979874Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8980250Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8980622Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8980990Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8981374Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8981756Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8982140Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8982519Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8982908Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8983315Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8983765Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8984157Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8984558Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8984952Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8985367Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8985773Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8986168Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8986536Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8986882Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8987252Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8987624Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8988005Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8988385Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8988821Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8989302Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8989737Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8990129Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8990528Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8990941Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8991376Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8991824Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8992253Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8992658Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8993082Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8993473Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8993827Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8994204Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8994590Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8994986Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8995402Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8995827Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8996286Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8996691Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8997137Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8997613Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8998163Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8998760Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8999355Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.8999955Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9000591Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9001183Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9001770Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9002357Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9002875Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9003355Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9003790Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9004162Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9004544Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9004932Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9005311Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9005710Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9006133Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9006574Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9007026Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9007422Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9007802Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9008202Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9008611Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9009025Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9009423Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9009822Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9010233Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9010733Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9011133Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9011588Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9012074Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9012657Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9013127Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9013619Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9014155Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9014657Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9015123Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9015559Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9016026Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9016512Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9016995Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9017487Z test_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9017908Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9018333Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9018766Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9019201Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9019629Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9020066Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9020509Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9020925Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9021332Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9021773Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9022228Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9022680Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9023100Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9023506Z test_isend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9023946Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9024404Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9024879Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9025409Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9025924Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9026401Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9026890Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9027395Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9027897Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9028387Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9028867Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9029359Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9029755Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9030146Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9030527Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9030913Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9031358Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9031821Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9032326Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9032759Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9033207Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9033643Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9034055Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9034457Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9034861Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9035283Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9035717Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9036188Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9036728Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9037198Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9037603Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9037976Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9038362Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9038749Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9039120Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9039485Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9039860Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9040224Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9040585Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9040930Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9041288Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9041650Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9042028Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9042412Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9042770Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9043124Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9043495Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9043863Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9044212Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9044557Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9044927Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9045284Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9045657Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9046034Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9046396Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9046756Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9047111Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9047470Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9047873Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9048303Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9048719Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9049149Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9049531Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9049941Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9050336Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9050809Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9051202Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9051622Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9052011Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9052392Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9052779Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9053159Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9053614Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9054020Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9054462Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9054884Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9055705Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9056346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9056783Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9057369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9057827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9058050Z 2022-11-23T02:06:51.9058147Z Running tests... 2022-11-23T02:06:51.9058555Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9059157Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3167 2022-11-23T02:06:51.9059758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3168 2022-11-23T02:06:51.9060250Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9060902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9061335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9061915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9062375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9062802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9063434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9063864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9064440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9064894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9065327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9065984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9066731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9067245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9067706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9068209Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:06:51.9069047Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:06:51.9069691Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:06:51.9070522Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:06:51.9071255Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9071755Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9072225Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:06:51.9073050Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:06:51.9073690Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:06:51.9074514Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:06:51.9075148Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:06:51.9075692Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:06:51.9076514Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:06:51.9077425Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:06:51.9078060Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:06:51.9078868Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:06:51.9079500Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:06:51.9080317Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:06:51.9080778Z ok (5.706s) 2022-11-23T02:06:51.9080914Z 2022-11-23T02:06:51.9081184Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9081494Z Ran 1 test in 5.707s 2022-11-23T02:06:51.9081640Z 2022-11-23T02:06:51.9081718Z OK 2022-11-23T02:06:51.9081839Z 2022-11-23T02:06:51.9081948Z Generating XML reports... 2022-11-23T02:06:51.9082546Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013757.xml 2022-11-23T02:06:51.9083198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9083883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9084327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9084909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9085364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9085579Z 2022-11-23T02:06:51.9085672Z Running tests... 2022-11-23T02:06:51.9086070Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9086556Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s) 2022-11-23T02:06:51.9086856Z 2022-11-23T02:06:51.9087118Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9087422Z Ran 1 test in 0.004s 2022-11-23T02:06:51.9087576Z 2022-11-23T02:06:51.9087671Z OK (skipped=1) 2022-11-23T02:06:51.9087871Z 2022-11-23T02:06:51.9087978Z Generating XML reports... 2022-11-23T02:06:51.9088585Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013807.xml 2022-11-23T02:06:51.9089227Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9089854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9090289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9091017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9091475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9091689Z 2022-11-23T02:06:51.9091781Z Running tests... 2022-11-23T02:06:51.9092186Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9092703Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3447 2022-11-23T02:06:51.9093233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3448 2022-11-23T02:06:51.9093729Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9094383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9094817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9095400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9095863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9096307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9096936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9097373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9097951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9098405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9098837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9099490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9100178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9100683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9101330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9101664Z ok (5.210s) 2022-11-23T02:06:51.9101796Z 2022-11-23T02:06:51.9102070Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9102384Z Ran 1 test in 5.210s 2022-11-23T02:06:51.9102530Z 2022-11-23T02:06:51.9102606Z OK 2022-11-23T02:06:51.9102725Z 2022-11-23T02:06:51.9102832Z Generating XML reports... 2022-11-23T02:06:51.9103422Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013811.xml 2022-11-23T02:06:51.9104071Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9104693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9105128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9105776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9106230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9106445Z 2022-11-23T02:06:51.9106536Z Running tests... 2022-11-23T02:06:51.9106944Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9108107Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.593s) 2022-11-23T02:06:51.9108731Z 2022-11-23T02:06:51.9108994Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9109293Z Ran 1 test in 0.593s 2022-11-23T02:06:51.9109444Z 2022-11-23T02:06:51.9109536Z OK (skipped=1) 2022-11-23T02:06:51.9109678Z 2022-11-23T02:06:51.9109785Z Generating XML reports... 2022-11-23T02:06:51.9110387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013821.xml 2022-11-23T02:06:51.9111033Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9111654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9112088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9112663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9113119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9113333Z 2022-11-23T02:06:51.9113422Z Running tests... 2022-11-23T02:06:51.9113824Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9114328Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.001s) 2022-11-23T02:06:51.9114641Z 2022-11-23T02:06:51.9114902Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9115207Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9115352Z 2022-11-23T02:06:51.9115435Z OK (skipped=1) 2022-11-23T02:06:51.9115578Z 2022-11-23T02:06:51.9115685Z Generating XML reports... 2022-11-23T02:06:51.9116289Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013826.xml 2022-11-23T02:06:51.9116932Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9117555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9117991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9118631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9119093Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9119302Z 2022-11-23T02:06:51.9119393Z Running tests... 2022-11-23T02:06:51.9119802Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9120318Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.001s) 2022-11-23T02:06:51.9120644Z 2022-11-23T02:06:51.9120909Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9121216Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9121363Z 2022-11-23T02:06:51.9121453Z OK (skipped=1) 2022-11-23T02:06:51.9121593Z 2022-11-23T02:06:51.9121700Z Generating XML reports... 2022-11-23T02:06:51.9122293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013830.xml 2022-11-23T02:06:51.9122990Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9123613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9124045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9124627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9125084Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9125297Z 2022-11-23T02:06:51.9125388Z Running tests... 2022-11-23T02:06:51.9125781Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9126345Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3848 2022-11-23T02:06:51.9126914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3849 2022-11-23T02:06:51.9127406Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9128060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9128494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9129075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9129528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9129954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9130702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9131406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9131994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9132448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9132880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9133539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9134221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9134727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9135186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9135738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptn_u9eq_ 2022-11-23T02:06:51.9136252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptn_u9eq_/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9136805Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9137261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptwum2xlh 2022-11-23T02:06:51.9137772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptwum2xlh/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9138317Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9139197Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9139781Z warnings.warn( 2022-11-23T02:06:51.9140139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9141038Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9141564Z warnings.warn( 2022-11-23T02:06:51.9141920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9142383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9142838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9143307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9143770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9144241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9144703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9145171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9145634Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9146091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9146559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9146890Z ok (9.038s) 2022-11-23T02:06:51.9147028Z 2022-11-23T02:06:51.9147300Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9147610Z Ran 1 test in 9.038s 2022-11-23T02:06:51.9147757Z 2022-11-23T02:06:51.9147840Z OK 2022-11-23T02:06:51.9147960Z 2022-11-23T02:06:51.9148063Z Generating XML reports... 2022-11-23T02:06:51.9148672Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013834.xml 2022-11-23T02:06:51.9149322Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9149950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9150387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9150984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9151445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9151660Z 2022-11-23T02:06:51.9151751Z Running tests... 2022-11-23T02:06:51.9152147Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9152774Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4068 2022-11-23T02:06:51.9153354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4069 2022-11-23T02:06:51.9153840Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9154503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9154944Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9155525Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9155974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9156406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9157089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9157522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9158106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9158560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9158996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9159661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9160337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9160847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9161312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9161794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocb6jkw3 2022-11-23T02:06:51.9162317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocb6jkw3/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9162824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9hktrq8 2022-11-23T02:06:51.9163339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9hktrq8/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9163885Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9164386Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9164868Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9165411Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9166323Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9166849Z warnings.warn( 2022-11-23T02:06:51.9167207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9168096Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9168613Z warnings.warn( 2022-11-23T02:06:51.9168961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9169481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9169947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9170283Z ok (5.651s) 2022-11-23T02:06:51.9170420Z 2022-11-23T02:06:51.9170771Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9171083Z Ran 1 test in 5.651s 2022-11-23T02:06:51.9171228Z 2022-11-23T02:06:51.9171296Z OK 2022-11-23T02:06:51.9171413Z 2022-11-23T02:06:51.9171520Z Generating XML reports... 2022-11-23T02:06:51.9172125Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013847.xml 2022-11-23T02:06:51.9172772Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9173397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9173906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9174497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9174959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9175167Z 2022-11-23T02:06:51.9175258Z Running tests... 2022-11-23T02:06:51.9175662Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9176234Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4286 2022-11-23T02:06:51.9176819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4287 2022-11-23T02:06:51.9177312Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9177975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9178416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9178983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9179439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9179876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9180508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9180937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9181517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9181975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9182400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9183050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9183737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9184238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9184699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9185180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgq1fo67q 2022-11-23T02:06:51.9185698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgq1fo67q/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9186314Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9186761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoyk5gui8 2022-11-23T02:06:51.9187276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoyk5gui8/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9187829Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9188686Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9189216Z warnings.warn( 2022-11-23T02:06:51.9189580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9190471Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9191055Z warnings.warn( 2022-11-23T02:06:51.9191403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9191868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9192330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9192791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9193261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9193722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9194183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9194508Z ok (5.719s) 2022-11-23T02:06:51.9194646Z 2022-11-23T02:06:51.9194923Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9195236Z Ran 1 test in 5.720s 2022-11-23T02:06:51.9195385Z 2022-11-23T02:06:51.9195461Z OK 2022-11-23T02:06:51.9195580Z 2022-11-23T02:06:51.9195688Z Generating XML reports... 2022-11-23T02:06:51.9196292Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013858.xml 2022-11-23T02:06:51.9196938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9197555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9197994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9198575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9199030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9199253Z 2022-11-23T02:06:51.9199348Z Running tests... 2022-11-23T02:06:51.9199751Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9200353Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4504 2022-11-23T02:06:51.9200944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4505 2022-11-23T02:06:51.9201433Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9202088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9202523Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9203180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9203646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9204081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9204718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9205143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9205720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9206170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9206600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9207251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9208001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9208506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9208969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9209437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvv1ztumo 2022-11-23T02:06:51.9209951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvv1ztumo/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9210457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2w4f8m3 2022-11-23T02:06:51.9211109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2w4f8m3/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9211665Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9212167Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9213027Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9213540Z warnings.warn( 2022-11-23T02:06:51.9213895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9214780Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9215303Z warnings.warn( 2022-11-23T02:06:51.9215653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9215989Z ok (5.917s) 2022-11-23T02:06:51.9216126Z 2022-11-23T02:06:51.9216396Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9216697Z Ran 1 test in 5.918s 2022-11-23T02:06:51.9216845Z 2022-11-23T02:06:51.9216922Z OK 2022-11-23T02:06:51.9217039Z 2022-11-23T02:06:51.9217148Z Generating XML reports... 2022-11-23T02:06:51.9217750Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013908.xml 2022-11-23T02:06:51.9218403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9219031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9219465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9220036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9220499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9220787Z 2022-11-23T02:06:51.9220881Z Running tests... 2022-11-23T02:06:51.9221288Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9221882Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4722 2022-11-23T02:06:51.9222479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4723 2022-11-23T02:06:51.9222972Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9223627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9224054Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9224642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9225156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9225587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9226231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9226663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9227240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9227694Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9228119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9228769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9229472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9229977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9230441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9230927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7f8ueugs 2022-11-23T02:06:51.9231439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7f8ueugs/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9231983Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9232438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptts96dzj 2022-11-23T02:06:51.9232949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptts96dzj/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9233506Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9234365Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9234887Z warnings.warn( 2022-11-23T02:06:51.9235243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9236123Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9236633Z warnings.warn( 2022-11-23T02:06:51.9236988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9237506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9237972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9238307Z ok (8.047s) 2022-11-23T02:06:51.9238441Z 2022-11-23T02:06:51.9238715Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9239025Z Ran 1 test in 8.047s 2022-11-23T02:06:51.9239171Z 2022-11-23T02:06:51.9239239Z OK 2022-11-23T02:06:51.9239355Z 2022-11-23T02:06:51.9239463Z Generating XML reports... 2022-11-23T02:06:51.9240067Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013918.xml 2022-11-23T02:06:51.9240718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9241341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9241833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9242425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9242871Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9243089Z 2022-11-23T02:06:51.9243180Z Running tests... 2022-11-23T02:06:51.9243583Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9244151Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4942 2022-11-23T02:06:51.9244740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4943 2022-11-23T02:06:51.9245237Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9245901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9246349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9246920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9247375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9247815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9248446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9248877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9249458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9249911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9250340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9251472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9252165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9252669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9253135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9253613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1143qkha 2022-11-23T02:06:51.9254123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1143qkha/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9254672Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9255197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa39ogpgk 2022-11-23T02:06:51.9255710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa39ogpgk/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9256261Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9257117Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9257640Z warnings.warn( 2022-11-23T02:06:51.9257998Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9258886Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9259466Z warnings.warn( 2022-11-23T02:06:51.9259814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9260278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9260737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9261069Z ok (8.732s) 2022-11-23T02:06:51.9261204Z 2022-11-23T02:06:51.9261485Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9261798Z Ran 1 test in 8.732s 2022-11-23T02:06:51.9261945Z 2022-11-23T02:06:51.9262013Z OK 2022-11-23T02:06:51.9262131Z 2022-11-23T02:06:51.9262241Z Generating XML reports... 2022-11-23T02:06:51.9262844Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013930.xml 2022-11-23T02:06:51.9263488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9264129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9264570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9265152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9265614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9265824Z 2022-11-23T02:06:51.9265918Z Running tests... 2022-11-23T02:06:51.9266339Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9266938Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5162 2022-11-23T02:06:51.9267544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5163 2022-11-23T02:06:51.9268040Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9268696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9269131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9269704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9270163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9270597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9271227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9271663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9272309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9272779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9273224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9273873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9274563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9275070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9275526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9276010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsu41v_y 2022-11-23T02:06:51.9276584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsu41v_y/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9277140Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9277588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph42ogu1g 2022-11-23T02:06:51.9278102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph42ogu1g/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9278658Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9279142Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9279679Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9280605Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9281135Z warnings.warn( 2022-11-23T02:06:51.9281498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9282369Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:51.9282896Z warnings.warn( 2022-11-23T02:06:51.9283252Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9283716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9284204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9284546Z ok (5.818s) 2022-11-23T02:06:51.9284681Z 2022-11-23T02:06:51.9284955Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9285256Z Ran 1 test in 5.818s 2022-11-23T02:06:51.9285403Z 2022-11-23T02:06:51.9285480Z OK 2022-11-23T02:06:51.9285596Z 2022-11-23T02:06:51.9285704Z Generating XML reports... 2022-11-23T02:06:51.9286313Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013943.xml 2022-11-23T02:06:51.9286964Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9287592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9288030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9288606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9289131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9289352Z 2022-11-23T02:06:51.9289445Z Running tests... 2022-11-23T02:06:51.9289853Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9291145Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.588s) 2022-11-23T02:06:51.9291778Z 2022-11-23T02:06:51.9292041Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9292351Z Ran 1 test in 0.589s 2022-11-23T02:06:51.9292498Z 2022-11-23T02:06:51.9292588Z OK (skipped=1) 2022-11-23T02:06:51.9292728Z 2022-11-23T02:06:51.9292837Z Generating XML reports... 2022-11-23T02:06:51.9293511Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013953.xml 2022-11-23T02:06:51.9294156Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9294784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9295225Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9295811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9296269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9296485Z 2022-11-23T02:06:51.9296578Z Running tests... 2022-11-23T02:06:51.9296972Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9297529Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5446 2022-11-23T02:06:51.9298101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5447 2022-11-23T02:06:51.9298588Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9299242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9299678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9300263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9300714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9301138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9301772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9302211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9302794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9303251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9303681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9304332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9305009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9305517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9306039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9306383Z ok (5.027s) 2022-11-23T02:06:51.9306521Z 2022-11-23T02:06:51.9306796Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9307109Z Ran 1 test in 5.027s 2022-11-23T02:06:51.9307258Z 2022-11-23T02:06:51.9307338Z OK 2022-11-23T02:06:51.9307459Z 2022-11-23T02:06:51.9307558Z Generating XML reports... 2022-11-23T02:06:51.9308159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013958.xml 2022-11-23T02:06:51.9308800Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9309424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9309858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9310446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9310963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9311177Z 2022-11-23T02:06:51.9311269Z Running tests... 2022-11-23T02:06:51.9311666Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9312868Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.587s) 2022-11-23T02:06:51.9313506Z 2022-11-23T02:06:51.9313770Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9314078Z Ran 1 test in 0.587s 2022-11-23T02:06:51.9314224Z 2022-11-23T02:06:51.9314321Z OK (skipped=1) 2022-11-23T02:06:51.9314461Z 2022-11-23T02:06:51.9314577Z Generating XML reports... 2022-11-23T02:06:51.9315177Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014007.xml 2022-11-23T02:06:51.9315821Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9316437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9316867Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9317456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9317915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9318131Z 2022-11-23T02:06:51.9318224Z Running tests... 2022-11-23T02:06:51.9318628Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9319180Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5715 2022-11-23T02:06:51.9319724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5716 2022-11-23T02:06:51.9320208Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9320868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9321304Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9321885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9322343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9322835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9323484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9323912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9324494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9324948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9325385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9326037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9326722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9327301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9327763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9328282Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9328780Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9329091Z ok (5.321s) 2022-11-23T02:06:51.9329225Z 2022-11-23T02:06:51.9329490Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9329800Z Ran 1 test in 5.321s 2022-11-23T02:06:51.9329945Z 2022-11-23T02:06:51.9330023Z OK 2022-11-23T02:06:51.9330140Z 2022-11-23T02:06:51.9330239Z Generating XML reports... 2022-11-23T02:06:51.9330990Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014012.xml 2022-11-23T02:06:51.9331646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9332285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9332719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9333306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9333764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9333979Z 2022-11-23T02:06:51.9334075Z Running tests... 2022-11-23T02:06:51.9334470Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9334926Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2022-11-23T02:06:51.9335201Z 2022-11-23T02:06:51.9335465Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9335771Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9335919Z 2022-11-23T02:06:51.9336009Z OK (skipped=1) 2022-11-23T02:06:51.9336152Z 2022-11-23T02:06:51.9336259Z Generating XML reports... 2022-11-23T02:06:51.9336848Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014021.xml 2022-11-23T02:06:51.9337493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9338111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9338554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9339134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9339590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9339808Z 2022-11-23T02:06:51.9339902Z Running tests... 2022-11-23T02:06:51.9340314Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9340794Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9341321Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5991 2022-11-23T02:06:51.9341842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5992 2022-11-23T02:06:51.9342338Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9342994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9343430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9344009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9344457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9344956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9345593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9346034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9346629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9347078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9347509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9348164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9348852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9349360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9349819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9350304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4rc7744l 2022-11-23T02:06:51.9350813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4rc7744l/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9351315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp44_mek5 2022-11-23T02:06:51.9351821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp44_mek5/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9352363Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9352859Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9353167Z ok (7.624s) 2022-11-23T02:06:51.9353303Z 2022-11-23T02:06:51.9353569Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9353878Z Ran 1 test in 7.624s 2022-11-23T02:06:51.9354023Z 2022-11-23T02:06:51.9354101Z OK 2022-11-23T02:06:51.9354218Z 2022-11-23T02:06:51.9354327Z Generating XML reports... 2022-11-23T02:06:51.9354917Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014025.xml 2022-11-23T02:06:51.9355563Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9356186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9356623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9357210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9357734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9357958Z 2022-11-23T02:06:51.9358053Z Running tests... 2022-11-23T02:06:51.9358450Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9358879Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9359424Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6211 2022-11-23T02:06:51.9359946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6212 2022-11-23T02:06:51.9360442Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9361109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9361605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9362182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9362638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9363074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9363706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9364136Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9364714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9365163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9365597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9366257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9366943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9367454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9367921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9368410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwz4lczy8 2022-11-23T02:06:51.9368925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwz4lczy8/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9369453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4mbc66l 2022-11-23T02:06:51.9369978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4mbc66l/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9370609Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9371142Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9371584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9372051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9372382Z ok (7.774s) 2022-11-23T02:06:51.9372517Z 2022-11-23T02:06:51.9372787Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9373090Z Ran 1 test in 7.774s 2022-11-23T02:06:51.9373239Z 2022-11-23T02:06:51.9373319Z OK 2022-11-23T02:06:51.9373436Z 2022-11-23T02:06:51.9373546Z Generating XML reports... 2022-11-23T02:06:51.9374146Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014037.xml 2022-11-23T02:06:51.9374871Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9375507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9375940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9376511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9376995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9377216Z 2022-11-23T02:06:51.9377307Z Running tests... 2022-11-23T02:06:51.9377711Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9378154Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9378732Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6431 2022-11-23T02:06:51.9379329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6432 2022-11-23T02:06:51.9379818Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9380469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9380904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9381536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9382052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9382511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9383169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9383641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9384234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9384716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9385150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9385801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9386488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9386991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9387449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9387938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb63tie0x 2022-11-23T02:06:51.9388439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb63tie0x/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9388988Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9389442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpihh_2zow 2022-11-23T02:06:51.9389952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpihh_2zow/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9390519Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9390969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9391434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:51.9391761Z ok (8.329s) 2022-11-23T02:06:51.9391892Z 2022-11-23T02:06:51.9392220Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9392539Z Ran 1 test in 8.330s 2022-11-23T02:06:51.9392688Z 2022-11-23T02:06:51.9392764Z OK 2022-11-23T02:06:51.9392881Z 2022-11-23T02:06:51.9392994Z Generating XML reports... 2022-11-23T02:06:51.9393599Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014049.xml 2022-11-23T02:06:51.9394259Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9394875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9395317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9395901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9396417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9396635Z 2022-11-23T02:06:51.9396732Z Running tests... 2022-11-23T02:06:51.9397148Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9397581Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:06:51.9398111Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6651 2022-11-23T02:06:51.9398646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6652 2022-11-23T02:06:51.9399147Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9399809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9400249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9400845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9401302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9401726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9402356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9402800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9403375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9403827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9404262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9404932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9405626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9406121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9406584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9407064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspn738zm 2022-11-23T02:06:51.9407583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspn738zm/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9408095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xn75687 2022-11-23T02:06:51.9408602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xn75687/_remote_module_non_scriptable.py 2022-11-23T02:06:51.9409215Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9409707Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9410013Z ok (7.733s) 2022-11-23T02:06:51.9410148Z 2022-11-23T02:06:51.9410413Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9410798Z Ran 1 test in 7.734s 2022-11-23T02:06:51.9410945Z 2022-11-23T02:06:51.9411023Z OK 2022-11-23T02:06:51.9411143Z 2022-11-23T02:06:51.9411250Z Generating XML reports... 2022-11-23T02:06:51.9411852Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014101.xml 2022-11-23T02:06:51.9412493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9413122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9413632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9414215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9414671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9414887Z 2022-11-23T02:06:51.9414977Z Running tests... 2022-11-23T02:06:51.9415386Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9415833Z test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9416108Z 2022-11-23T02:06:51.9416373Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9416681Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9416827Z 2022-11-23T02:06:51.9416919Z OK (skipped=1) 2022-11-23T02:06:51.9417058Z 2022-11-23T02:06:51.9417166Z Generating XML reports... 2022-11-23T02:06:51.9417775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014113.xml 2022-11-23T02:06:51.9418424Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9419042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9419482Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9420062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9420514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9420728Z 2022-11-23T02:06:51.9420820Z Running tests... 2022-11-23T02:06:51.9421225Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9421725Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:06:51.9422041Z 2022-11-23T02:06:51.9422310Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9422611Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9422762Z 2022-11-23T02:06:51.9422853Z OK (skipped=1) 2022-11-23T02:06:51.9422995Z 2022-11-23T02:06:51.9423103Z Generating XML reports... 2022-11-23T02:06:51.9423700Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014117.xml 2022-11-23T02:06:51.9424346Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9424973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9425412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9425988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9426522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9426747Z 2022-11-23T02:06:51.9426848Z Running tests... 2022-11-23T02:06:51.9427253Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9427756Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s) 2022-11-23T02:06:51.9428072Z 2022-11-23T02:06:51.9428335Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9428641Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9428647Z 2022-11-23T02:06:51.9428739Z OK (skipped=1) 2022-11-23T02:06:51.9428745Z 2022-11-23T02:06:51.9428852Z Generating XML reports... 2022-11-23T02:06:51.9429282Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014121.xml 2022-11-23T02:06:51.9429598Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9430093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9430257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9430649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9430827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9430833Z 2022-11-23T02:06:51.9430929Z Running tests... 2022-11-23T02:06:51.9431195Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9431473Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s) 2022-11-23T02:06:51.9431479Z 2022-11-23T02:06:51.9431740Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9431842Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9431848Z 2022-11-23T02:06:51.9431946Z OK (skipped=1) 2022-11-23T02:06:51.9431951Z 2022-11-23T02:06:51.9432060Z Generating XML reports... 2022-11-23T02:06:51.9432505Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014125.xml 2022-11-23T02:06:51.9432819Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9433198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9433361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9433747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9433927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9433933Z 2022-11-23T02:06:51.9434024Z Running tests... 2022-11-23T02:06:51.9434299Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9434581Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s) 2022-11-23T02:06:51.9434587Z 2022-11-23T02:06:51.9434852Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9434940Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9434955Z 2022-11-23T02:06:51.9435037Z OK (skipped=1) 2022-11-23T02:06:51.9435043Z 2022-11-23T02:06:51.9435151Z Generating XML reports... 2022-11-23T02:06:51.9435593Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014130.xml 2022-11-23T02:06:51.9435907Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9436281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9436495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9436888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9437068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9437074Z 2022-11-23T02:06:51.9437168Z Running tests... 2022-11-23T02:06:51.9437430Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9437715Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s) 2022-11-23T02:06:51.9437721Z 2022-11-23T02:06:51.9437979Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9438080Z Ran 1 test in 0.003s 2022-11-23T02:06:51.9438086Z 2022-11-23T02:06:51.9438179Z OK (skipped=1) 2022-11-23T02:06:51.9438184Z 2022-11-23T02:06:51.9438292Z Generating XML reports... 2022-11-23T02:06:51.9438788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014134.xml 2022-11-23T02:06:51.9439103Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9439480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9439641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9440024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9440199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9440205Z 2022-11-23T02:06:51.9440300Z Running tests... 2022-11-23T02:06:51.9440554Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9440810Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:51.9440821Z 2022-11-23T02:06:51.9441080Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9441175Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9441181Z 2022-11-23T02:06:51.9441271Z OK (skipped=1) 2022-11-23T02:06:51.9441277Z 2022-11-23T02:06:51.9441385Z Generating XML reports... 2022-11-23T02:06:51.9441827Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014138.xml 2022-11-23T02:06:51.9442144Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9442523Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9442687Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9443069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9443254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9443260Z 2022-11-23T02:06:51.9443354Z Running tests... 2022-11-23T02:06:51.9443618Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9443916Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7333 2022-11-23T02:06:51.9444119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7334 2022-11-23T02:06:51.9444377Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9444751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9444913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9445378Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9445564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9445794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9446161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9446326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9446713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9446887Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9447116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9447524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9447974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9448195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9448412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9448758Z STAGE:2022-11-23 01:41:45 7333:7333 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9449041Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9449373Z STAGE:2022-11-23 01:41:45 7334:7334 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9449654Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9449990Z STAGE:2022-11-23 01:41:45 7333:7333 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9450339Z STAGE:2022-11-23 01:41:45 7334:7334 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9450828Z STAGE:2022-11-23 01:41:45 7333:7333 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9451186Z STAGE:2022-11-23 01:41:45 7334:7334 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9451412Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9451634Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9451964Z STAGE:2022-11-23 01:41:45 7333:7333 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9452297Z STAGE:2022-11-23 01:41:45 7333:7333 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9452644Z STAGE:2022-11-23 01:41:45 7333:7333 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9452978Z STAGE:2022-11-23 01:41:45 7334:7334 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9453301Z STAGE:2022-11-23 01:41:45 7334:7334 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9453649Z STAGE:2022-11-23 01:41:45 7334:7334 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9453736Z ok (5.524s) 2022-11-23T02:06:51.9453742Z 2022-11-23T02:06:51.9454009Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9454105Z Ran 1 test in 5.524s 2022-11-23T02:06:51.9454110Z 2022-11-23T02:06:51.9454186Z OK 2022-11-23T02:06:51.9454192Z 2022-11-23T02:06:51.9454298Z Generating XML reports... 2022-11-23T02:06:51.9454739Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014142.xml 2022-11-23T02:06:51.9455051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9455514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9455676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9456061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9456237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9456243Z 2022-11-23T02:06:51.9456334Z Running tests... 2022-11-23T02:06:51.9456598Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9456911Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7549 2022-11-23T02:06:51.9457114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7550 2022-11-23T02:06:51.9457425Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9457805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9457970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9458365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9458545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9458759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9459136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9459299Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9459688Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9459867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9460088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9460491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9460886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9461101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9461320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9461652Z STAGE:2022-11-23 01:41:54 7550:7550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9461990Z STAGE:2022-11-23 01:41:55 7549:7549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9462265Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9462538Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9462873Z STAGE:2022-11-23 01:41:55 7550:7550 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9463202Z STAGE:2022-11-23 01:41:55 7549:7549 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9463552Z STAGE:2022-11-23 01:41:55 7550:7550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9463900Z STAGE:2022-11-23 01:41:55 7549:7549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9464124Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9464394Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9464731Z STAGE:2022-11-23 01:41:55 7549:7549 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9465060Z STAGE:2022-11-23 01:41:55 7550:7550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9465383Z STAGE:2022-11-23 01:41:55 7549:7549 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9465732Z STAGE:2022-11-23 01:41:55 7549:7549 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9466065Z STAGE:2022-11-23 01:41:55 7550:7550 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9466413Z STAGE:2022-11-23 01:41:55 7550:7550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9466496Z ok (5.420s) 2022-11-23T02:06:51.9466502Z 2022-11-23T02:06:51.9466767Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9466910Z Ran 1 test in 5.420s 2022-11-23T02:06:51.9466920Z 2022-11-23T02:06:51.9466998Z OK 2022-11-23T02:06:51.9467003Z 2022-11-23T02:06:51.9467111Z Generating XML reports... 2022-11-23T02:06:51.9467556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014152.xml 2022-11-23T02:06:51.9467869Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9468245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9468404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9468793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9468969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9468975Z 2022-11-23T02:06:51.9469073Z Running tests... 2022-11-23T02:06:51.9469340Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9469599Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:51.9469605Z 2022-11-23T02:06:51.9469868Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9469962Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9469968Z 2022-11-23T02:06:51.9470060Z OK (skipped=1) 2022-11-23T02:06:51.9470065Z 2022-11-23T02:06:51.9470173Z Generating XML reports... 2022-11-23T02:06:51.9470608Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014201.xml 2022-11-23T02:06:51.9470919Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9471292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9471466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9471850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9472027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9472033Z 2022-11-23T02:06:51.9472124Z Running tests... 2022-11-23T02:06:51.9472389Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9472637Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:51.9472643Z 2022-11-23T02:06:51.9472902Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9472997Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9473002Z 2022-11-23T02:06:51.9473093Z OK (skipped=1) 2022-11-23T02:06:51.9473098Z 2022-11-23T02:06:51.9473207Z Generating XML reports... 2022-11-23T02:06:51.9473709Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014205.xml 2022-11-23T02:06:51.9474028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9474400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9474562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9474953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9475131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9475136Z 2022-11-23T02:06:51.9475230Z Running tests... 2022-11-23T02:06:51.9475498Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9475818Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7897 2022-11-23T02:06:51.9476069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7898 2022-11-23T02:06:51.9476318Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9476698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9476864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9477257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9477437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9477666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9478045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9478213Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9478602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9478782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9479008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9479412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9479810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9480032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9480249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9480597Z STAGE:2022-11-23 01:42:12 7897:7897 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9480875Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9481205Z STAGE:2022-11-23 01:42:12 7898:7898 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9481478Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9481817Z STAGE:2022-11-23 01:42:12 7897:7897 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9482157Z STAGE:2022-11-23 01:42:12 7898:7898 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9482508Z STAGE:2022-11-23 01:42:12 7897:7897 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9482846Z STAGE:2022-11-23 01:42:12 7898:7898 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9483123Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9483353Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9483448Z ok (5.224s) 2022-11-23T02:06:51.9483453Z 2022-11-23T02:06:51.9483727Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9483825Z Ran 1 test in 5.225s 2022-11-23T02:06:51.9483830Z 2022-11-23T02:06:51.9483911Z OK 2022-11-23T02:06:51.9483917Z 2022-11-23T02:06:51.9484034Z Generating XML reports... 2022-11-23T02:06:51.9484480Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014209.xml 2022-11-23T02:06:51.9484799Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9485178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9485391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9485783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9485965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9485971Z 2022-11-23T02:06:51.9486069Z Running tests... 2022-11-23T02:06:51.9486339Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9486665Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8113 2022-11-23T02:06:51.9486871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8114 2022-11-23T02:06:51.9487138Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9487521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9487682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9488067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9488243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9488459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9488830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9488991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9489377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9489553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9489786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9490188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9490694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9490912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9491123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9491457Z STAGE:2022-11-23 01:42:22 8113:8113 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9491733Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9492062Z STAGE:2022-11-23 01:42:22 8114:8114 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9492408Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9492751Z STAGE:2022-11-23 01:42:22 8114:8114 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9493085Z STAGE:2022-11-23 01:42:22 8113:8113 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9493438Z STAGE:2022-11-23 01:42:22 8114:8114 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9493789Z STAGE:2022-11-23 01:42:22 8113:8113 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9494011Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9494233Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9494321Z ok (5.419s) 2022-11-23T02:06:51.9494327Z 2022-11-23T02:06:51.9494654Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9494742Z Ran 1 test in 5.419s 2022-11-23T02:06:51.9494747Z 2022-11-23T02:06:51.9494825Z OK 2022-11-23T02:06:51.9494831Z 2022-11-23T02:06:51.9494938Z Generating XML reports... 2022-11-23T02:06:51.9495382Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014219.xml 2022-11-23T02:06:51.9495697Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9496073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9496237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9496624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9496802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9496812Z 2022-11-23T02:06:51.9496909Z Running tests... 2022-11-23T02:06:51.9497178Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9498095Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.585s) 2022-11-23T02:06:51.9498102Z 2022-11-23T02:06:51.9498370Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9498464Z Ran 1 test in 0.585s 2022-11-23T02:06:51.9498470Z 2022-11-23T02:06:51.9498560Z OK (skipped=1) 2022-11-23T02:06:51.9498565Z 2022-11-23T02:06:51.9498673Z Generating XML reports... 2022-11-23T02:06:51.9499118Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014228.xml 2022-11-23T02:06:51.9499436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9499809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9499968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9500351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9500529Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9500535Z 2022-11-23T02:06:51.9500626Z Running tests... 2022-11-23T02:06:51.9500891Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9501874Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.585s) 2022-11-23T02:06:51.9501886Z 2022-11-23T02:06:51.9502149Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9502246Z Ran 1 test in 0.585s 2022-11-23T02:06:51.9502252Z 2022-11-23T02:06:51.9502347Z OK (skipped=1) 2022-11-23T02:06:51.9502352Z 2022-11-23T02:06:51.9502451Z Generating XML reports... 2022-11-23T02:06:51.9502894Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014233.xml 2022-11-23T02:06:51.9503209Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9503583Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9503793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9504181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9504355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9504361Z 2022-11-23T02:06:51.9504451Z Running tests... 2022-11-23T02:06:51.9504715Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9505032Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8461 2022-11-23T02:06:51.9505236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8462 2022-11-23T02:06:51.9505489Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9505865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9506031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9506413Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9506588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9506815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9507186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9507344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9507726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9507905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9508135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9508525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9508919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9509134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9509344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9509618Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9509892Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9509978Z ok (5.511s) 2022-11-23T02:06:51.9509984Z 2022-11-23T02:06:51.9510249Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9510398Z Ran 1 test in 5.512s 2022-11-23T02:06:51.9510404Z 2022-11-23T02:06:51.9510483Z OK 2022-11-23T02:06:51.9510489Z 2022-11-23T02:06:51.9510596Z Generating XML reports... 2022-11-23T02:06:51.9511044Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014237.xml 2022-11-23T02:06:51.9511358Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9511729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9511890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9512273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9512448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9512515Z 2022-11-23T02:06:51.9512609Z Running tests... 2022-11-23T02:06:51.9512879Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9513197Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8671 2022-11-23T02:06:51.9513403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8672 2022-11-23T02:06:51.9513658Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9514021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9514182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9514564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9514739Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9514969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9515340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9515500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9515883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9516058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9516279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9516675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9517073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9517291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9517509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9517735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:06:51.9517958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:06:51.9518364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:51.9518761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:51.9519039Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9519313Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9519586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:06:51.9519984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:06:51.9520208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:06:51.9520601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:06:51.9520814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T02:06:51.9521205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:06:51.9521425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T02:06:51.9521873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:06:51.9521957Z ok (6.110s) 2022-11-23T02:06:51.9521963Z 2022-11-23T02:06:51.9522228Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9522321Z Ran 1 test in 6.110s 2022-11-23T02:06:51.9522327Z 2022-11-23T02:06:51.9522402Z OK 2022-11-23T02:06:51.9522408Z 2022-11-23T02:06:51.9522516Z Generating XML reports... 2022-11-23T02:06:51.9522963Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014247.xml 2022-11-23T02:06:51.9523282Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9523660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9523826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9524221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9524403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9524409Z 2022-11-23T02:06:51.9524502Z Running tests... 2022-11-23T02:06:51.9524769Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9525072Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8916 2022-11-23T02:06:51.9525276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8917 2022-11-23T02:06:51.9525532Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9525906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9526076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9526461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9526628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9526856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9527231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9527394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9527778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9527959Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9528190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9528647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9529057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9529275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9529487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9529765Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9530045Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9530135Z ok (5.829s) 2022-11-23T02:06:51.9530141Z 2022-11-23T02:06:51.9530403Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9530502Z Ran 1 test in 5.829s 2022-11-23T02:06:51.9530732Z 2022-11-23T02:06:51.9530818Z OK 2022-11-23T02:06:51.9530824Z 2022-11-23T02:06:51.9530937Z Generating XML reports... 2022-11-23T02:06:51.9531390Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014257.xml 2022-11-23T02:06:51.9531703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9532076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9532228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9532611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9532787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9532792Z 2022-11-23T02:06:51.9532885Z Running tests... 2022-11-23T02:06:51.9533157Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9533593Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:06:51.9533599Z 2022-11-23T02:06:51.9533864Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9533961Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9533967Z 2022-11-23T02:06:51.9534062Z OK (skipped=1) 2022-11-23T02:06:51.9534068Z 2022-11-23T02:06:51.9534180Z Generating XML reports... 2022-11-23T02:06:51.9534622Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014307.xml 2022-11-23T02:06:51.9534936Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9535310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9535481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9535867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9536042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9536047Z 2022-11-23T02:06:51.9536138Z Running tests... 2022-11-23T02:06:51.9536404Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9536833Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:51.9536840Z 2022-11-23T02:06:51.9537098Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9537196Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9537201Z 2022-11-23T02:06:51.9537292Z OK (skipped=1) 2022-11-23T02:06:51.9537298Z 2022-11-23T02:06:51.9537406Z Generating XML reports... 2022-11-23T02:06:51.9537908Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014311.xml 2022-11-23T02:06:51.9538228Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9538600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9538761Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9539146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9539320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9539326Z 2022-11-23T02:06:51.9539422Z Running tests... 2022-11-23T02:06:51.9539687Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9540132Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:51.9540193Z 2022-11-23T02:06:51.9540453Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9540547Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9540552Z 2022-11-23T02:06:51.9540642Z OK (skipped=1) 2022-11-23T02:06:51.9540648Z 2022-11-23T02:06:51.9540759Z Generating XML reports... 2022-11-23T02:06:51.9541200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014315.xml 2022-11-23T02:06:51.9541516Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9541891Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9542052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9542441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9542622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9542628Z 2022-11-23T02:06:51.9542720Z Running tests... 2022-11-23T02:06:51.9542982Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9543407Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.001s) 2022-11-23T02:06:51.9543413Z 2022-11-23T02:06:51.9543681Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9543781Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9543786Z 2022-11-23T02:06:51.9543869Z OK (skipped=1) 2022-11-23T02:06:51.9543885Z 2022-11-23T02:06:51.9543985Z Generating XML reports... 2022-11-23T02:06:51.9544429Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014319.xml 2022-11-23T02:06:51.9544750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9545125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9545286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9545671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9545845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9545850Z 2022-11-23T02:06:51.9545943Z Running tests... 2022-11-23T02:06:51.9546207Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9546625Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:51.9546631Z 2022-11-23T02:06:51.9546891Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9547045Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9547051Z 2022-11-23T02:06:51.9547141Z OK (skipped=1) 2022-11-23T02:06:51.9547147Z 2022-11-23T02:06:51.9547253Z Generating XML reports... 2022-11-23T02:06:51.9547694Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014323.xml 2022-11-23T02:06:51.9548010Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9548386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9548550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9548936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9549112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9549170Z 2022-11-23T02:06:51.9549261Z Running tests... 2022-11-23T02:06:51.9549524Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9549944Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:51.9549960Z 2022-11-23T02:06:51.9550213Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9550312Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9550318Z 2022-11-23T02:06:51.9550408Z OK (skipped=1) 2022-11-23T02:06:51.9550413Z 2022-11-23T02:06:51.9550523Z Generating XML reports... 2022-11-23T02:06:51.9550958Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014327.xml 2022-11-23T02:06:51.9551274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9551649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9551815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9552201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9552379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9552385Z 2022-11-23T02:06:51.9552477Z Running tests... 2022-11-23T02:06:51.9552741Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9553168Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:51.9553174Z 2022-11-23T02:06:51.9553437Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9553534Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9553540Z 2022-11-23T02:06:51.9553630Z OK (skipped=1) 2022-11-23T02:06:51.9553639Z 2022-11-23T02:06:51.9553748Z Generating XML reports... 2022-11-23T02:06:51.9554192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014331.xml 2022-11-23T02:06:51.9554508Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9554881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9555042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9555426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9555592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9555606Z 2022-11-23T02:06:51.9555689Z Running tests... 2022-11-23T02:06:51.9555955Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9556435Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:51.9556442Z 2022-11-23T02:06:51.9556707Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9556802Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9556807Z 2022-11-23T02:06:51.9556898Z OK (skipped=1) 2022-11-23T02:06:51.9556904Z 2022-11-23T02:06:51.9557011Z Generating XML reports... 2022-11-23T02:06:51.9557453Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014335.xml 2022-11-23T02:06:51.9557770Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9558143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9558305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9558692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9558916Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9558922Z 2022-11-23T02:06:51.9559013Z Running tests... 2022-11-23T02:06:51.9559277Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9559686Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:51.9559692Z 2022-11-23T02:06:51.9559953Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9560048Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9560053Z 2022-11-23T02:06:51.9560143Z OK (skipped=1) 2022-11-23T02:06:51.9560148Z 2022-11-23T02:06:51.9560255Z Generating XML reports... 2022-11-23T02:06:51.9560699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014339.xml 2022-11-23T02:06:51.9561019Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9561385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9561548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9561935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9562108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9562114Z 2022-11-23T02:06:51.9562206Z Running tests... 2022-11-23T02:06:51.9562468Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9562760Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9562766Z 2022-11-23T02:06:51.9563034Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9563133Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9563139Z 2022-11-23T02:06:51.9563231Z OK (skipped=1) 2022-11-23T02:06:51.9563237Z 2022-11-23T02:06:51.9563346Z Generating XML reports... 2022-11-23T02:06:51.9563786Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014343.xml 2022-11-23T02:06:51.9564103Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9564474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9564635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9565017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9565190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9565199Z 2022-11-23T02:06:51.9565340Z Running tests... 2022-11-23T02:06:51.9565607Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9566011Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:51.9566017Z 2022-11-23T02:06:51.9566280Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9566377Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9566383Z 2022-11-23T02:06:51.9566480Z OK (skipped=1) 2022-11-23T02:06:51.9566485Z 2022-11-23T02:06:51.9566584Z Generating XML reports... 2022-11-23T02:06:51.9567023Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014347.xml 2022-11-23T02:06:51.9567341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9567720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9567930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9568317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9568492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9568498Z 2022-11-23T02:06:51.9568588Z Running tests... 2022-11-23T02:06:51.9568854Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9569270Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:51.9569277Z 2022-11-23T02:06:51.9569537Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9569631Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9569637Z 2022-11-23T02:06:51.9569731Z OK (skipped=1) 2022-11-23T02:06:51.9569737Z 2022-11-23T02:06:51.9569849Z Generating XML reports... 2022-11-23T02:06:51.9570285Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014351.xml 2022-11-23T02:06:51.9570728Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9571106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9571267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9571649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9571826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9571832Z 2022-11-23T02:06:51.9571927Z Running tests... 2022-11-23T02:06:51.9572190Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9572596Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:51.9572610Z 2022-11-23T02:06:51.9572862Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9572955Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9572961Z 2022-11-23T02:06:51.9573051Z OK (skipped=1) 2022-11-23T02:06:51.9573057Z 2022-11-23T02:06:51.9573164Z Generating XML reports... 2022-11-23T02:06:51.9573606Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014355.xml 2022-11-23T02:06:51.9573920Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9574301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9574461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9574911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9575089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9575095Z 2022-11-23T02:06:51.9575188Z Running tests... 2022-11-23T02:06:51.9575454Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9575729Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9575735Z 2022-11-23T02:06:51.9575995Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9576090Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9576096Z 2022-11-23T02:06:51.9576190Z OK (skipped=1) 2022-11-23T02:06:51.9576196Z 2022-11-23T02:06:51.9576302Z Generating XML reports... 2022-11-23T02:06:51.9576741Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014359.xml 2022-11-23T02:06:51.9577112Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9577484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9577648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9578034Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9578208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9578214Z 2022-11-23T02:06:51.9578298Z Running tests... 2022-11-23T02:06:51.9578582Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9578848Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:51.9578878Z 2022-11-23T02:06:51.9579132Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9579229Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9579235Z 2022-11-23T02:06:51.9579326Z OK (skipped=1) 2022-11-23T02:06:51.9579332Z 2022-11-23T02:06:51.9579438Z Generating XML reports... 2022-11-23T02:06:51.9579878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014403.xml 2022-11-23T02:06:51.9580193Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9580565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9580731Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9581116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9581299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9581311Z 2022-11-23T02:06:51.9581409Z Running tests... 2022-11-23T02:06:51.9581672Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9581929Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:51.9581935Z 2022-11-23T02:06:51.9582200Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9582296Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9582302Z 2022-11-23T02:06:51.9582394Z OK (skipped=1) 2022-11-23T02:06:51.9582399Z 2022-11-23T02:06:51.9582508Z Generating XML reports... 2022-11-23T02:06:51.9582946Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014407.xml 2022-11-23T02:06:51.9583260Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9583702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9583869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9584257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9584432Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9584438Z 2022-11-23T02:06:51.9584521Z Running tests... 2022-11-23T02:06:51.9584786Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9585058Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9585064Z 2022-11-23T02:06:51.9585324Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9585421Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9585426Z 2022-11-23T02:06:51.9585565Z OK (skipped=1) 2022-11-23T02:06:51.9585571Z 2022-11-23T02:06:51.9585683Z Generating XML reports... 2022-11-23T02:06:51.9586130Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014411.xml 2022-11-23T02:06:51.9586444Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9586818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9586982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9587367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9587544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9587550Z 2022-11-23T02:06:51.9587643Z Running tests... 2022-11-23T02:06:51.9587910Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9588177Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9588183Z 2022-11-23T02:06:51.9588445Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9588546Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9588552Z 2022-11-23T02:06:51.9588644Z OK (skipped=1) 2022-11-23T02:06:51.9588650Z 2022-11-23T02:06:51.9588757Z Generating XML reports... 2022-11-23T02:06:51.9589197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014415.xml 2022-11-23T02:06:51.9589512Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9589884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9590037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9590430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9590609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9590615Z 2022-11-23T02:06:51.9590707Z Running tests... 2022-11-23T02:06:51.9590971Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9591226Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9591232Z 2022-11-23T02:06:51.9591494Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9591588Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9591594Z 2022-11-23T02:06:51.9591684Z OK (skipped=1) 2022-11-23T02:06:51.9591689Z 2022-11-23T02:06:51.9591799Z Generating XML reports... 2022-11-23T02:06:51.9592248Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014419.xml 2022-11-23T02:06:51.9592621Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9593006Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9593168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9593553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9593729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9593735Z 2022-11-23T02:06:51.9593827Z Running tests... 2022-11-23T02:06:51.9594096Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9594350Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9594356Z 2022-11-23T02:06:51.9594672Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9594772Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9594778Z 2022-11-23T02:06:51.9594870Z OK (skipped=1) 2022-11-23T02:06:51.9594875Z 2022-11-23T02:06:51.9594975Z Generating XML reports... 2022-11-23T02:06:51.9595419Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014423.xml 2022-11-23T02:06:51.9595737Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9596114Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9596276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9596666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9596848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9596858Z 2022-11-23T02:06:51.9596954Z Running tests... 2022-11-23T02:06:51.9597220Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9597485Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9597491Z 2022-11-23T02:06:51.9597752Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9597850Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9597855Z 2022-11-23T02:06:51.9597947Z OK (skipped=1) 2022-11-23T02:06:51.9597953Z 2022-11-23T02:06:51.9598068Z Generating XML reports... 2022-11-23T02:06:51.9598505Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014428.xml 2022-11-23T02:06:51.9598824Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9599208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9599376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9599761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9599944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9599950Z 2022-11-23T02:06:51.9600043Z Running tests... 2022-11-23T02:06:51.9600308Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9600552Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9600568Z 2022-11-23T02:06:51.9600818Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9600915Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9600921Z 2022-11-23T02:06:51.9601010Z OK (skipped=1) 2022-11-23T02:06:51.9601020Z 2022-11-23T02:06:51.9601128Z Generating XML reports... 2022-11-23T02:06:51.9601623Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014432.xml 2022-11-23T02:06:51.9601947Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9602324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9602485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9602868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9603045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9603050Z 2022-11-23T02:06:51.9603142Z Running tests... 2022-11-23T02:06:51.9603411Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9603717Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9603724Z 2022-11-23T02:06:51.9603989Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9604090Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9604095Z 2022-11-23T02:06:51.9604191Z OK (skipped=1) 2022-11-23T02:06:51.9604196Z 2022-11-23T02:06:51.9604304Z Generating XML reports... 2022-11-23T02:06:51.9604746Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014436.xml 2022-11-23T02:06:51.9605059Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9605433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9605594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9605983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9606155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9606171Z 2022-11-23T02:06:51.9606254Z Running tests... 2022-11-23T02:06:51.9606518Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9606768Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:51.9606774Z 2022-11-23T02:06:51.9607036Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9607131Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9607137Z 2022-11-23T02:06:51.9607229Z OK (skipped=1) 2022-11-23T02:06:51.9607235Z 2022-11-23T02:06:51.9607346Z Generating XML reports... 2022-11-23T02:06:51.9607792Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014440.xml 2022-11-23T02:06:51.9608116Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9608492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9608654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9609041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9609219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9609225Z 2022-11-23T02:06:51.9609319Z Running tests... 2022-11-23T02:06:51.9609583Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9609847Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s) 2022-11-23T02:06:51.9609853Z 2022-11-23T02:06:51.9610116Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9610215Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9610293Z 2022-11-23T02:06:51.9610387Z OK (skipped=1) 2022-11-23T02:06:51.9610392Z 2022-11-23T02:06:51.9610501Z Generating XML reports... 2022-11-23T02:06:51.9611031Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014444.xml 2022-11-23T02:06:51.9611348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9611711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9611871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9612261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9612443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9612507Z 2022-11-23T02:06:51.9612600Z Running tests... 2022-11-23T02:06:51.9612869Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9613155Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s) 2022-11-23T02:06:51.9613162Z 2022-11-23T02:06:51.9613427Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9613527Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9613532Z 2022-11-23T02:06:51.9613630Z OK (skipped=1) 2022-11-23T02:06:51.9613635Z 2022-11-23T02:06:51.9613748Z Generating XML reports... 2022-11-23T02:06:51.9614198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014448.xml 2022-11-23T02:06:51.9614517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9614896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9615066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9615456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9615641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9615647Z 2022-11-23T02:06:51.9615745Z Running tests... 2022-11-23T02:06:51.9616013Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9616272Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9616278Z 2022-11-23T02:06:51.9616543Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9616645Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9616650Z 2022-11-23T02:06:51.9616746Z OK (skipped=1) 2022-11-23T02:06:51.9616752Z 2022-11-23T02:06:51.9616851Z Generating XML reports... 2022-11-23T02:06:51.9617306Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014452.xml 2022-11-23T02:06:51.9617626Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9618006Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9618167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9618558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9618740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9618746Z 2022-11-23T02:06:51.9618841Z Running tests... 2022-11-23T02:06:51.9619109Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9619482Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10912 2022-11-23T02:06:51.9619700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10913 2022-11-23T02:06:51.9619957Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9620341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9620506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9620894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9621072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9621302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9621680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9621890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9622281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9622458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9622690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9623092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9623479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9623694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9623917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9624198Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9624481Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9624571Z ok (5.308s) 2022-11-23T02:06:51.9624577Z 2022-11-23T02:06:51.9624841Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9624943Z Ran 1 test in 5.309s 2022-11-23T02:06:51.9624949Z 2022-11-23T02:06:51.9625027Z OK 2022-11-23T02:06:51.9625032Z 2022-11-23T02:06:51.9625143Z Generating XML reports... 2022-11-23T02:06:51.9625592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014456.xml 2022-11-23T02:06:51.9625911Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9626290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9626457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9626839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9627015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9627021Z 2022-11-23T02:06:51.9627121Z Running tests... 2022-11-23T02:06:51.9627387Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9627638Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9627644Z 2022-11-23T02:06:51.9627909Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9628008Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9628014Z 2022-11-23T02:06:51.9628096Z OK (skipped=1) 2022-11-23T02:06:51.9628120Z 2022-11-23T02:06:51.9628222Z Generating XML reports... 2022-11-23T02:06:51.9628720Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014505.xml 2022-11-23T02:06:51.9629046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9629425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9629593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9629979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9630161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9630167Z 2022-11-23T02:06:51.9630265Z Running tests... 2022-11-23T02:06:51.9630535Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9630796Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9630853Z 2022-11-23T02:06:51.9631126Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9631226Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9631232Z 2022-11-23T02:06:51.9631327Z OK (skipped=1) 2022-11-23T02:06:51.9631333Z 2022-11-23T02:06:51.9631446Z Generating XML reports... 2022-11-23T02:06:51.9631890Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014509.xml 2022-11-23T02:06:51.9632208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9632585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9632753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9633150Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9633332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9633338Z 2022-11-23T02:06:51.9633437Z Running tests... 2022-11-23T02:06:51.9633703Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9633952Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9633972Z 2022-11-23T02:06:51.9634224Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9634324Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9634330Z 2022-11-23T02:06:51.9634425Z OK (skipped=1) 2022-11-23T02:06:51.9634431Z 2022-11-23T02:06:51.9634543Z Generating XML reports... 2022-11-23T02:06:51.9634989Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014513.xml 2022-11-23T02:06:51.9635318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9635698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9635863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9636251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9636427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9636433Z 2022-11-23T02:06:51.9636529Z Running tests... 2022-11-23T02:06:51.9636796Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9637107Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11320 2022-11-23T02:06:51.9637317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11321 2022-11-23T02:06:51.9637628Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9638014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9638177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9638564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9638744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9638972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9639351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9639520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9639942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9640122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9640349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9640752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9641152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9641372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9641595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9641939Z STAGE:2022-11-23 01:45:20 11320:11320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9642227Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9642558Z STAGE:2022-11-23 01:45:20 11321:11321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9642833Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9643180Z STAGE:2022-11-23 01:45:20 11321:11321 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9643757Z STAGE:2022-11-23 01:45:20 11320:11320 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:20 11321:11321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9643763Z 2022-11-23T02:06:51.9643996Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9644357Z STAGE:2022-11-23 01:45:20 11320:11320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9644591Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9644929Z STAGE:2022-11-23 01:45:20 11320:11320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9645265Z STAGE:2022-11-23 01:45:20 11321:11321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9645608Z STAGE:2022-11-23 01:45:20 11320:11320 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9645967Z STAGE:2022-11-23 01:45:20 11320:11320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9646308Z STAGE:2022-11-23 01:45:20 11321:11321 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9646664Z STAGE:2022-11-23 01:45:20 11321:11321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9647000Z STAGE:2022-11-23 01:45:20 11320:11320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9647389Z STAGE:2022-11-23 01:45:20 11321:11321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9647733Z STAGE:2022-11-23 01:45:20 11321:11321 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9648079Z STAGE:2022-11-23 01:45:20 11321:11321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9648421Z STAGE:2022-11-23 01:45:20 11320:11320 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9648775Z STAGE:2022-11-23 01:45:20 11320:11320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9648865Z ok (5.535s) 2022-11-23T02:06:51.9648871Z 2022-11-23T02:06:51.9649140Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9649243Z Ran 1 test in 5.536s 2022-11-23T02:06:51.9649249Z 2022-11-23T02:06:51.9649331Z OK 2022-11-23T02:06:51.9649337Z 2022-11-23T02:06:51.9649514Z Generating XML reports... 2022-11-23T02:06:51.9649970Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014517.xml 2022-11-23T02:06:51.9650285Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9650790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9650954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9651343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9651521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9651527Z 2022-11-23T02:06:51.9651620Z Running tests... 2022-11-23T02:06:51.9651888Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9652208Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11536 2022-11-23T02:06:51.9652423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11537 2022-11-23T02:06:51.9652677Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9653050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9653211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9653597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9653764Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9653988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9654367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9654528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9654911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9655086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9655312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9655711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9656104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9656318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9656602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9656945Z STAGE:2022-11-23 01:45:30 11536:11536 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9657220Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9657553Z STAGE:2022-11-23 01:45:30 11537:11537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9657827Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9658164Z STAGE:2022-11-23 01:45:30 11537:11537 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9658498Z STAGE:2022-11-23 01:45:30 11536:11536 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9658851Z STAGE:2022-11-23 01:45:30 11537:11537 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9659205Z STAGE:2022-11-23 01:45:30 11536:11536 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9659480Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9659702Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9660040Z STAGE:2022-11-23 01:45:30 11537:11537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9660370Z STAGE:2022-11-23 01:45:30 11536:11536 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9660698Z STAGE:2022-11-23 01:45:30 11537:11537 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9661048Z STAGE:2022-11-23 01:45:30 11537:11537 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9661383Z STAGE:2022-11-23 01:45:30 11536:11536 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9661743Z STAGE:2022-11-23 01:45:30 11536:11536 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9662072Z STAGE:2022-11-23 01:45:30 11536:11536 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9662401Z STAGE:2022-11-23 01:45:30 11537:11537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9662735Z STAGE:2022-11-23 01:45:30 11536:11536 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9663072Z STAGE:2022-11-23 01:45:30 11537:11537 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9663424Z STAGE:2022-11-23 01:45:30 11536:11536 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9663770Z STAGE:2022-11-23 01:45:30 11537:11537 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9663858Z ok (5.334s) 2022-11-23T02:06:51.9663864Z 2022-11-23T02:06:51.9664128Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9664230Z Ran 1 test in 5.335s 2022-11-23T02:06:51.9664236Z 2022-11-23T02:06:51.9664314Z OK 2022-11-23T02:06:51.9664320Z 2022-11-23T02:06:51.9664427Z Generating XML reports... 2022-11-23T02:06:51.9664871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014527.xml 2022-11-23T02:06:51.9665185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9665559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9665721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9666108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9666283Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9666292Z 2022-11-23T02:06:51.9666382Z Running tests... 2022-11-23T02:06:51.9666687Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9667008Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11752 2022-11-23T02:06:51.9667217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11753 2022-11-23T02:06:51.9667475Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9667852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9668013Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9668398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9668620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9668848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9669225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9669386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9669770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9669952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9670179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9670585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9670987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9671207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9671420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9671756Z STAGE:2022-11-23 01:45:39 11753:11753 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9672085Z STAGE:2022-11-23 01:45:39 11752:11752 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9672360Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9672634Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9672971Z STAGE:2022-11-23 01:45:39 11753:11753 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9673295Z STAGE:2022-11-23 01:45:39 11752:11752 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9673656Z STAGE:2022-11-23 01:45:39 11753:11753 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9674008Z STAGE:2022-11-23 01:45:39 11752:11752 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9674234Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9674459Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9674791Z STAGE:2022-11-23 01:45:39 11753:11753 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9675119Z STAGE:2022-11-23 01:45:39 11752:11752 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9675456Z STAGE:2022-11-23 01:45:39 11753:11753 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9675808Z STAGE:2022-11-23 01:45:39 11753:11753 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9676201Z STAGE:2022-11-23 01:45:39 11752:11752 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9676555Z STAGE:2022-11-23 01:45:39 11752:11752 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9676885Z STAGE:2022-11-23 01:45:39 11753:11753 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9677211Z STAGE:2022-11-23 01:45:39 11752:11752 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9677546Z STAGE:2022-11-23 01:45:39 11753:11753 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9677899Z STAGE:2022-11-23 01:45:39 11753:11753 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9678233Z STAGE:2022-11-23 01:45:39 11752:11752 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9678585Z STAGE:2022-11-23 01:45:39 11752:11752 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9678718Z ok (5.812s) 2022-11-23T02:06:51.9678724Z 2022-11-23T02:06:51.9678990Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9679088Z Ran 1 test in 5.812s 2022-11-23T02:06:51.9679094Z 2022-11-23T02:06:51.9679172Z OK 2022-11-23T02:06:51.9679178Z 2022-11-23T02:06:51.9679290Z Generating XML reports... 2022-11-23T02:06:51.9679738Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014536.xml 2022-11-23T02:06:51.9680047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9680429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9680601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9680994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9681178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9681184Z 2022-11-23T02:06:51.9681280Z Running tests... 2022-11-23T02:06:51.9681554Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9681788Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:06:51.9681794Z 2022-11-23T02:06:51.9682061Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9682163Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9682168Z 2022-11-23T02:06:51.9682265Z OK (skipped=1) 2022-11-23T02:06:51.9682270Z 2022-11-23T02:06:51.9682384Z Generating XML reports... 2022-11-23T02:06:51.9682830Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014546.xml 2022-11-23T02:06:51.9683158Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9683536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9683703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9684093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9684273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9684279Z 2022-11-23T02:06:51.9684381Z Running tests... 2022-11-23T02:06:51.9684650Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9684899Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:06:51.9684905Z 2022-11-23T02:06:51.9685169Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9685260Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9685332Z 2022-11-23T02:06:51.9685417Z OK (skipped=1) 2022-11-23T02:06:51.9685438Z 2022-11-23T02:06:51.9685538Z Generating XML reports... 2022-11-23T02:06:51.9685987Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014550.xml 2022-11-23T02:06:51.9686306Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9686685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9686856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9687238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9687420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9687426Z 2022-11-23T02:06:51.9687570Z Running tests... 2022-11-23T02:06:51.9687844Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9688143Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12100 2022-11-23T02:06:51.9688353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12101 2022-11-23T02:06:51.9688612Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9688994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9689160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9689550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9689731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9689965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9690341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9690506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9690979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9691167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9691384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9691789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9692191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9692421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9692639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9692768Z skip: Test skipped for ROCm (5.412s) 2022-11-23T02:06:51.9692774Z 2022-11-23T02:06:51.9693048Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9693146Z Ran 1 test in 5.412s 2022-11-23T02:06:51.9693152Z 2022-11-23T02:06:51.9693249Z OK (skipped=1) 2022-11-23T02:06:51.9693255Z 2022-11-23T02:06:51.9693368Z Generating XML reports... 2022-11-23T02:06:51.9693816Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014554.xml 2022-11-23T02:06:51.9694133Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9694569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9694743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9695136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9695320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9695326Z 2022-11-23T02:06:51.9695421Z Running tests... 2022-11-23T02:06:51.9695691Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9696004Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12303 2022-11-23T02:06:51.9696213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12304 2022-11-23T02:06:51.9696474Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9696912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9697077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9697451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9697630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9697859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9698236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9698399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9698786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9698975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9699203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9699605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9700004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9700224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9700443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9700571Z skip: Test skipped for ROCm (5.652s) 2022-11-23T02:06:51.9700578Z 2022-11-23T02:06:51.9700846Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9700952Z Ran 1 test in 5.653s 2022-11-23T02:06:51.9700962Z 2022-11-23T02:06:51.9701059Z OK (skipped=1) 2022-11-23T02:06:51.9701065Z 2022-11-23T02:06:51.9701178Z Generating XML reports... 2022-11-23T02:06:51.9701628Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014604.xml 2022-11-23T02:06:51.9701947Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9702323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9702492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9702884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9703051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9703071Z 2022-11-23T02:06:51.9703154Z Running tests... 2022-11-23T02:06:51.9703478Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9703731Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:06:51.9703737Z 2022-11-23T02:06:51.9704007Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9704106Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9704112Z 2022-11-23T02:06:51.9704209Z OK (skipped=1) 2022-11-23T02:06:51.9704214Z 2022-11-23T02:06:51.9704330Z Generating XML reports... 2022-11-23T02:06:51.9704774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014614.xml 2022-11-23T02:06:51.9705089Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9705466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9705630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9706088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9706273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9706279Z 2022-11-23T02:06:51.9706376Z Running tests... 2022-11-23T02:06:51.9706644Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9706961Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12572 2022-11-23T02:06:51.9707170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12573 2022-11-23T02:06:51.9707431Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9707808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9707980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9708368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9708550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9708768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9709143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9709309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9709699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9709878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9710115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9710520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9710930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9711150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9711371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9711500Z skip: Test skipped for ROCm (5.533s) 2022-11-23T02:06:51.9711506Z 2022-11-23T02:06:51.9711774Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9711874Z Ran 1 test in 5.533s 2022-11-23T02:06:51.9711879Z 2022-11-23T02:06:51.9711976Z OK (skipped=1) 2022-11-23T02:06:51.9711981Z 2022-11-23T02:06:51.9712096Z Generating XML reports... 2022-11-23T02:06:51.9712596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014618.xml 2022-11-23T02:06:51.9712921Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9713300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9713466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9713855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9714037Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9714042Z 2022-11-23T02:06:51.9714138Z Running tests... 2022-11-23T02:06:51.9714394Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9714641Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:06:51.9714708Z 2022-11-23T02:06:51.9714964Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9715068Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9715074Z 2022-11-23T02:06:51.9715169Z OK (skipped=1) 2022-11-23T02:06:51.9715174Z 2022-11-23T02:06:51.9715289Z Generating XML reports... 2022-11-23T02:06:51.9715735Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014627.xml 2022-11-23T02:06:51.9716053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9716428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9716597Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9716988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9717182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9717188Z 2022-11-23T02:06:51.9717288Z Running tests... 2022-11-23T02:06:51.9717555Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9717865Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12841 2022-11-23T02:06:51.9718076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12842 2022-11-23T02:06:51.9718338Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9718716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9718883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9719274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9719454Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9719684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9720063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9720216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9720603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9720783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9721009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9721475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9721888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9722109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9722326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9722476Z skip: Skipped due to small world size. (4.815s) 2022-11-23T02:06:51.9722482Z 2022-11-23T02:06:51.9722750Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9722850Z Ran 1 test in 4.815s 2022-11-23T02:06:51.9722856Z 2022-11-23T02:06:51.9722953Z OK (skipped=1) 2022-11-23T02:06:51.9722959Z 2022-11-23T02:06:51.9723073Z Generating XML reports... 2022-11-23T02:06:51.9723517Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014631.xml 2022-11-23T02:06:51.9723890Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9724268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9724434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9724823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9725004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9725010Z 2022-11-23T02:06:51.9725107Z Running tests... 2022-11-23T02:06:51.9725379Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9725654Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:06:51.9725663Z 2022-11-23T02:06:51.9725934Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9726021Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9726040Z 2022-11-23T02:06:51.9726122Z OK (skipped=1) 2022-11-23T02:06:51.9726128Z 2022-11-23T02:06:51.9726239Z Generating XML reports... 2022-11-23T02:06:51.9726690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014640.xml 2022-11-23T02:06:51.9727007Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9727383Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9727554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9727945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9728125Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9728137Z 2022-11-23T02:06:51.9728233Z Running tests... 2022-11-23T02:06:51.9728496Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9728790Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-11-23T02:06:51.9728796Z 2022-11-23T02:06:51.9729059Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9729160Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9729165Z 2022-11-23T02:06:51.9729263Z OK (skipped=1) 2022-11-23T02:06:51.9729269Z 2022-11-23T02:06:51.9729381Z Generating XML reports... 2022-11-23T02:06:51.9729827Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014645.xml 2022-11-23T02:06:51.9730146Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9730657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9730831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9731226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9731408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9731414Z 2022-11-23T02:06:51.9731512Z Running tests... 2022-11-23T02:06:51.9731767Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9732722Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.595s) 2022-11-23T02:06:51.9732798Z 2022-11-23T02:06:51.9733070Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9733157Z Ran 1 test in 0.596s 2022-11-23T02:06:51.9733177Z 2022-11-23T02:06:51.9733260Z OK (skipped=1) 2022-11-23T02:06:51.9733265Z 2022-11-23T02:06:51.9733378Z Generating XML reports... 2022-11-23T02:06:51.9733827Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014649.xml 2022-11-23T02:06:51.9734149Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9734530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9734694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9735079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9735265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9735271Z 2022-11-23T02:06:51.9735369Z Running tests... 2022-11-23T02:06:51.9735635Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9735970Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13242 2022-11-23T02:06:51.9736180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13243 2022-11-23T02:06:51.9736440Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9736821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9736986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9737380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9737563Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9737795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9738169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9738333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9738723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9738904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9739120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9739569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9739980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9740202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9740424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9740764Z STAGE:2022-11-23 01:46:56 13242:13242 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9741045Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9741389Z STAGE:2022-11-23 01:46:56 13243:13243 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:51.9741670Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9742014Z STAGE:2022-11-23 01:46:57 13243:13243 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9742429Z STAGE:2022-11-23 01:46:57 13243:13243 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9742659Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9743001Z STAGE:2022-11-23 01:46:57 13242:13242 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:51.9743354Z STAGE:2022-11-23 01:46:57 13242:13242 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:51.9743578Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:51.9743668Z ok (5.914s) 2022-11-23T02:06:51.9743674Z 2022-11-23T02:06:51.9743942Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9744042Z Ran 1 test in 5.914s 2022-11-23T02:06:51.9744048Z 2022-11-23T02:06:51.9744132Z OK 2022-11-23T02:06:51.9744143Z 2022-11-23T02:06:51.9744259Z Generating XML reports... 2022-11-23T02:06:51.9744708Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014653.xml 2022-11-23T02:06:51.9745028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9745390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9745559Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9745948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9746129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9746135Z 2022-11-23T02:06:51.9746232Z Running tests... 2022-11-23T02:06:51.9746499Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9746794Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:06:51.9746800Z 2022-11-23T02:06:51.9747065Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9747165Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9747171Z 2022-11-23T02:06:51.9747268Z OK (skipped=1) 2022-11-23T02:06:51.9747274Z 2022-11-23T02:06:51.9747390Z Generating XML reports... 2022-11-23T02:06:51.9747835Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014704.xml 2022-11-23T02:06:51.9748157Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9748540Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9748706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9749156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9749340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9749346Z 2022-11-23T02:06:51.9749444Z Running tests... 2022-11-23T02:06:51.9749714Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9750687Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.580s) 2022-11-23T02:06:51.9750694Z 2022-11-23T02:06:51.9750959Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9751062Z Ran 1 test in 0.581s 2022-11-23T02:06:51.9751113Z 2022-11-23T02:06:51.9751207Z OK (skipped=1) 2022-11-23T02:06:51.9751216Z 2022-11-23T02:06:51.9751327Z Generating XML reports... 2022-11-23T02:06:51.9751777Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014708.xml 2022-11-23T02:06:51.9752099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9752463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9752628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9753020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9753203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9753209Z 2022-11-23T02:06:51.9753305Z Running tests... 2022-11-23T02:06:51.9753574Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9753864Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-11-23T02:06:51.9753870Z 2022-11-23T02:06:51.9754136Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9754235Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9754241Z 2022-11-23T02:06:51.9754337Z OK (skipped=1) 2022-11-23T02:06:51.9754343Z 2022-11-23T02:06:51.9754457Z Generating XML reports... 2022-11-23T02:06:51.9754899Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014712.xml 2022-11-23T02:06:51.9755215Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9755596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9755762Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9756157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9756338Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9756345Z 2022-11-23T02:06:51.9756442Z Running tests... 2022-11-23T02:06:51.9756714Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9757050Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13660 2022-11-23T02:06:51.9757260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13661 2022-11-23T02:06:51.9757522Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9757903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9758114Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9758507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9758692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9758923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9759300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9759466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9759852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9760034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9760312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9760721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9761125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9761346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9761565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9761713Z skip: Skipped due to small world size. (5.317s) 2022-11-23T02:06:51.9761719Z 2022-11-23T02:06:51.9761989Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9762090Z Ran 1 test in 5.318s 2022-11-23T02:06:51.9762096Z 2022-11-23T02:06:51.9762194Z OK (skipped=1) 2022-11-23T02:06:51.9762204Z 2022-11-23T02:06:51.9762317Z Generating XML reports... 2022-11-23T02:06:51.9762767Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014716.xml 2022-11-23T02:06:51.9763085Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9763465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9763632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9764022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9764188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9764211Z 2022-11-23T02:06:51.9764294Z Running tests... 2022-11-23T02:06:51.9764564Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9764853Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:06:51.9764859Z 2022-11-23T02:06:51.9765125Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9765226Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9765231Z 2022-11-23T02:06:51.9765329Z OK (skipped=1) 2022-11-23T02:06:51.9765334Z 2022-11-23T02:06:51.9765446Z Generating XML reports... 2022-11-23T02:06:51.9765891Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014726.xml 2022-11-23T02:06:51.9766213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9766593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9766759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9767203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9767391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9767397Z 2022-11-23T02:06:51.9767495Z Running tests... 2022-11-23T02:06:51.9767765Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9768060Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:06:51.9768066Z 2022-11-23T02:06:51.9768335Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9768441Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9768447Z 2022-11-23T02:06:51.9768541Z OK (skipped=1) 2022-11-23T02:06:51.9768547Z 2022-11-23T02:06:51.9768664Z Generating XML reports... 2022-11-23T02:06:51.9769110Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014730.xml 2022-11-23T02:06:51.9769484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9769852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9770017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9770407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9770727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9770734Z 2022-11-23T02:06:51.9770837Z Running tests... 2022-11-23T02:06:51.9771110Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9771449Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13995 2022-11-23T02:06:51.9771669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13996 2022-11-23T02:06:51.9771933Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9772313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9772478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9772869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9773050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9773279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9773656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9773826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9774216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9774397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9774626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9775030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9775433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9775651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9775870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9776208Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9776492Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9776584Z ok (5.510s) 2022-11-23T02:06:51.9776590Z 2022-11-23T02:06:51.9776858Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9776956Z Ran 1 test in 5.510s 2022-11-23T02:06:51.9776961Z 2022-11-23T02:06:51.9777042Z OK 2022-11-23T02:06:51.9777048Z 2022-11-23T02:06:51.9777161Z Generating XML reports... 2022-11-23T02:06:51.9777609Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014734.xml 2022-11-23T02:06:51.9777930Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9778308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9778475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9778938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9779118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9779124Z 2022-11-23T02:06:51.9779225Z Running tests... 2022-11-23T02:06:51.9779493Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9779837Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14209 2022-11-23T02:06:51.9780049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14210 2022-11-23T02:06:51.9780315Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9780694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9780868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9781256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9781434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9781652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9782028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9782197Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9782586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9782769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9783005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9783404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9783806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9784026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9784241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9784522Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9784802Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9784895Z ok (5.617s) 2022-11-23T02:06:51.9784901Z 2022-11-23T02:06:51.9785165Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9785270Z Ran 1 test in 5.618s 2022-11-23T02:06:51.9785323Z 2022-11-23T02:06:51.9785407Z OK 2022-11-23T02:06:51.9785413Z 2022-11-23T02:06:51.9785527Z Generating XML reports... 2022-11-23T02:06:51.9785976Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014743.xml 2022-11-23T02:06:51.9786294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9786671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9786836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9787211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9787395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9787414Z 2022-11-23T02:06:51.9787580Z Running tests... 2022-11-23T02:06:51.9787855Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9788152Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:06:51.9788158Z 2022-11-23T02:06:51.9788427Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9788526Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9788531Z 2022-11-23T02:06:51.9788636Z OK (skipped=1) 2022-11-23T02:06:51.9788641Z 2022-11-23T02:06:51.9788756Z Generating XML reports... 2022-11-23T02:06:51.9789198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014753.xml 2022-11-23T02:06:51.9789516Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9789895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9790069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9790457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9790641Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9790646Z 2022-11-23T02:06:51.9790747Z Running tests... 2022-11-23T02:06:51.9791019Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9791368Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14489 2022-11-23T02:06:51.9791579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14490 2022-11-23T02:06:51.9791840Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9792223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9792390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9792779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9792961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9793177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9793556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9793720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9794109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9794303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9794583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9794987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9795386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9795601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9795826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:06:51.9796049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9796277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:06:51.9796734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:51.9797140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:51.9797419Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9797696Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9797788Z ok (5.514s) 2022-11-23T02:06:51.9797794Z 2022-11-23T02:06:51.9798061Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9798162Z Ran 1 test in 5.514s 2022-11-23T02:06:51.9798168Z 2022-11-23T02:06:51.9798252Z OK 2022-11-23T02:06:51.9798258Z 2022-11-23T02:06:51.9798371Z Generating XML reports... 2022-11-23T02:06:51.9798819Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014757.xml 2022-11-23T02:06:51.9799131Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9799507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9799673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9800065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9800245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9800251Z 2022-11-23T02:06:51.9800349Z Running tests... 2022-11-23T02:06:51.9800617Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9800910Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:06:51.9800917Z 2022-11-23T02:06:51.9801182Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9801289Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9801295Z 2022-11-23T02:06:51.9801391Z OK (skipped=1) 2022-11-23T02:06:51.9801397Z 2022-11-23T02:06:51.9801511Z Generating XML reports... 2022-11-23T02:06:51.9801957Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014807.xml 2022-11-23T02:06:51.9802274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9802648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9802817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9803211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9803391Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9803399Z 2022-11-23T02:06:51.9803495Z Running tests... 2022-11-23T02:06:51.9803811Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9804151Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14771 2022-11-23T02:06:51.9804370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14772 2022-11-23T02:06:51.9804632Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9805000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9805165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9805553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9805781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9806010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9806399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9806566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9806953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9807133Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9807361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9807764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9808173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9808395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9808614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9808765Z skip: Skipped due to small world size. (5.408s) 2022-11-23T02:06:51.9808771Z 2022-11-23T02:06:51.9809044Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9809144Z Ran 1 test in 5.408s 2022-11-23T02:06:51.9809150Z 2022-11-23T02:06:51.9809247Z OK (skipped=1) 2022-11-23T02:06:51.9809252Z 2022-11-23T02:06:51.9809372Z Generating XML reports... 2022-11-23T02:06:51.9809820Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014811.xml 2022-11-23T02:06:51.9810137Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9810599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9810768Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9811146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9811326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9811331Z 2022-11-23T02:06:51.9811426Z Running tests... 2022-11-23T02:06:51.9811693Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9812005Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14974 2022-11-23T02:06:51.9812215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14975 2022-11-23T02:06:51.9812544Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9812942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9813108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9813497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9813678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9813905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9814281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9814446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9814839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9815070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9815300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9815709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9816112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9816333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9816555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9816837Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9817100Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9817336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:06:51.9817572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:06:51.9817975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:51.9818374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:51.9818470Z ok (5.610s) 2022-11-23T02:06:51.9818476Z 2022-11-23T02:06:51.9818743Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9818842Z Ran 1 test in 5.610s 2022-11-23T02:06:51.9818847Z 2022-11-23T02:06:51.9818931Z OK 2022-11-23T02:06:51.9818937Z 2022-11-23T02:06:51.9819049Z Generating XML reports... 2022-11-23T02:06:51.9819508Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014820.xml 2022-11-23T02:06:51.9819827Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9820206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9820373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9820762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9820943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9820949Z 2022-11-23T02:06:51.9821046Z Running tests... 2022-11-23T02:06:51.9821310Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9821746Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.009s) 2022-11-23T02:06:51.9821757Z 2022-11-23T02:06:51.9822076Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9822180Z Ran 1 test in 0.009s 2022-11-23T02:06:51.9822186Z 2022-11-23T02:06:51.9822280Z OK (skipped=1) 2022-11-23T02:06:51.9822285Z 2022-11-23T02:06:51.9822404Z Generating XML reports... 2022-11-23T02:06:51.9822839Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014830.xml 2022-11-23T02:06:51.9823164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9823547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9823713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9824102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9824336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9824341Z 2022-11-23T02:06:51.9824438Z Running tests... 2022-11-23T02:06:51.9824710Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9825134Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:06:51.9825141Z 2022-11-23T02:06:51.9825405Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9825505Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9825511Z 2022-11-23T02:06:51.9825605Z OK (skipped=1) 2022-11-23T02:06:51.9825611Z 2022-11-23T02:06:51.9825725Z Generating XML reports... 2022-11-23T02:06:51.9826173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014834.xml 2022-11-23T02:06:51.9826499Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9826893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9827058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9827447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9827626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9827632Z 2022-11-23T02:06:51.9827730Z Running tests... 2022-11-23T02:06:51.9827999Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9828242Z test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.001s) 2022-11-23T02:06:51.9828248Z 2022-11-23T02:06:51.9828514Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9828602Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9828626Z 2022-11-23T02:06:51.9828710Z OK (skipped=1) 2022-11-23T02:06:51.9828716Z 2022-11-23T02:06:51.9828829Z Generating XML reports... 2022-11-23T02:06:51.9829278Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014838.xml 2022-11-23T02:06:51.9829599Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9829980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9830146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9830532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9830712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9830718Z 2022-11-23T02:06:51.9830816Z Running tests... 2022-11-23T02:06:51.9831144Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9831449Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15395 2022-11-23T02:06:51.9831660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15396 2022-11-23T02:06:51.9831922Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9832304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9832472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9832857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9833042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9833325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9833707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9833874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9834265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9834434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9834661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9835067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9835471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9835701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9835921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9836200Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9836478Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9836567Z ok (6.410s) 2022-11-23T02:06:51.9836574Z 2022-11-23T02:06:51.9836846Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9836948Z Ran 1 test in 6.411s 2022-11-23T02:06:51.9836953Z 2022-11-23T02:06:51.9837037Z OK 2022-11-23T02:06:51.9837043Z 2022-11-23T02:06:51.9837154Z Generating XML reports... 2022-11-23T02:06:51.9837601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014842.xml 2022-11-23T02:06:51.9837919Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9838302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9838471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9838861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9839041Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9839047Z 2022-11-23T02:06:51.9839145Z Running tests... 2022-11-23T02:06:51.9839411Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9839669Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.001s) 2022-11-23T02:06:51.9839676Z 2022-11-23T02:06:51.9839943Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9840034Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9840040Z 2022-11-23T02:06:51.9840189Z OK (skipped=1) 2022-11-23T02:06:51.9840196Z 2022-11-23T02:06:51.9840311Z Generating XML reports... 2022-11-23T02:06:51.9840764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014853.xml 2022-11-23T02:06:51.9841084Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9841470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9841635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9842023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9842204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9842209Z 2022-11-23T02:06:51.9842305Z Running tests... 2022-11-23T02:06:51.9842630Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9842950Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15671 2022-11-23T02:06:51.9843165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15672 2022-11-23T02:06:51.9843422Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9843801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9843969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9844359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9844538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9844778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9845159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9845326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9845713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9845878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9846105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9846510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9846913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9847142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9847359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9847511Z skip: Skipped due to small world size. (4.912s) 2022-11-23T02:06:51.9847517Z 2022-11-23T02:06:51.9847789Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9847889Z Ran 1 test in 4.912s 2022-11-23T02:06:51.9847895Z 2022-11-23T02:06:51.9847992Z OK (skipped=1) 2022-11-23T02:06:51.9847997Z 2022-11-23T02:06:51.9848110Z Generating XML reports... 2022-11-23T02:06:51.9848553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014857.xml 2022-11-23T02:06:51.9848874Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9849306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9849480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9849872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9850055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9850061Z 2022-11-23T02:06:51.9850155Z Running tests... 2022-11-23T02:06:51.9850425Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9850759Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s) 2022-11-23T02:06:51.9850765Z 2022-11-23T02:06:51.9851047Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9851153Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9851159Z 2022-11-23T02:06:51.9851242Z OK (skipped=1) 2022-11-23T02:06:51.9851260Z 2022-11-23T02:06:51.9851510Z Generating XML reports... 2022-11-23T02:06:51.9851968Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014906.xml 2022-11-23T02:06:51.9852284Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9852662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9852828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9853218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9853401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9853407Z 2022-11-23T02:06:51.9853504Z Running tests... 2022-11-23T02:06:51.9853778Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9854093Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15940 2022-11-23T02:06:51.9854309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15941 2022-11-23T02:06:51.9854567Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9854948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9855115Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9855505Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9855687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9855917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9856303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9856478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9856869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9857050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9857277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9857666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9858065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9858281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9858581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9858739Z skip: Skipped due to small world size. (4.907s) 2022-11-23T02:06:51.9858745Z 2022-11-23T02:06:51.9859018Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9859118Z Ran 1 test in 4.908s 2022-11-23T02:06:51.9859124Z 2022-11-23T02:06:51.9859219Z OK (skipped=1) 2022-11-23T02:06:51.9859225Z 2022-11-23T02:06:51.9859339Z Generating XML reports... 2022-11-23T02:06:51.9859785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014910.xml 2022-11-23T02:06:51.9860099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9860478Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9860644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9861086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9861266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9861273Z 2022-11-23T02:06:51.9861370Z Running tests... 2022-11-23T02:06:51.9861640Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9861911Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s) 2022-11-23T02:06:51.9861917Z 2022-11-23T02:06:51.9862183Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9862286Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9862291Z 2022-11-23T02:06:51.9862390Z OK (skipped=1) 2022-11-23T02:06:51.9862395Z 2022-11-23T02:06:51.9862508Z Generating XML reports... 2022-11-23T02:06:51.9862958Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014919.xml 2022-11-23T02:06:51.9863269Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9863650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9863818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9864208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9864388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9864394Z 2022-11-23T02:06:51.9864489Z Running tests... 2022-11-23T02:06:51.9864758Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9865027Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s) 2022-11-23T02:06:51.9865035Z 2022-11-23T02:06:51.9865304Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9865407Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9865413Z 2022-11-23T02:06:51.9865512Z OK (skipped=1) 2022-11-23T02:06:51.9865518Z 2022-11-23T02:06:51.9865632Z Generating XML reports... 2022-11-23T02:06:51.9866080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014923.xml 2022-11-23T02:06:51.9866401Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9866780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9866946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9867334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9867569Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9867575Z 2022-11-23T02:06:51.9867675Z Running tests... 2022-11-23T02:06:51.9867946Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9868211Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s) 2022-11-23T02:06:51.9868217Z 2022-11-23T02:06:51.9868485Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9868586Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9868592Z 2022-11-23T02:06:51.9868674Z OK (skipped=1) 2022-11-23T02:06:51.9868696Z 2022-11-23T02:06:51.9868795Z Generating XML reports... 2022-11-23T02:06:51.9869241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014927.xml 2022-11-23T02:06:51.9869558Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9870001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9870166Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9870555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9870740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9870746Z 2022-11-23T02:06:51.9870845Z Running tests... 2022-11-23T02:06:51.9871116Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9871363Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s) 2022-11-23T02:06:51.9871369Z 2022-11-23T02:06:51.9871634Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9871736Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9871742Z 2022-11-23T02:06:51.9871843Z OK (skipped=1) 2022-11-23T02:06:51.9871849Z 2022-11-23T02:06:51.9871965Z Generating XML reports... 2022-11-23T02:06:51.9872411Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014931.xml 2022-11-23T02:06:51.9872732Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9873110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9873274Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9873664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9873847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9873853Z 2022-11-23T02:06:51.9873949Z Running tests... 2022-11-23T02:06:51.9874202Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9874459Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s) 2022-11-23T02:06:51.9874465Z 2022-11-23T02:06:51.9874733Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9874835Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9874841Z 2022-11-23T02:06:51.9874938Z OK (skipped=1) 2022-11-23T02:06:51.9874943Z 2022-11-23T02:06:51.9875054Z Generating XML reports... 2022-11-23T02:06:51.9875500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014935.xml 2022-11-23T02:06:51.9875826Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9876207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9876372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9876819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9877004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9877010Z 2022-11-23T02:06:51.9877112Z Running tests... 2022-11-23T02:06:51.9877384Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9877718Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16473 2022-11-23T02:06:51.9877929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16474 2022-11-23T02:06:51.9878187Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9878571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9878788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9879183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9879365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9879597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9879979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9880133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9880518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9880700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9880932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9881344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9881743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9881966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9882186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9882416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:06:51.9882645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:06:51.9883044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:51.9883278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:06:51.9883680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:51.9883904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:06:51.9884302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:06:51.9884704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:06:51.9884793Z ok (5.011s) 2022-11-23T02:06:51.9884799Z 2022-11-23T02:06:51.9885071Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9885171Z Ran 1 test in 5.011s 2022-11-23T02:06:51.9885177Z 2022-11-23T02:06:51.9885259Z OK 2022-11-23T02:06:51.9885268Z 2022-11-23T02:06:51.9885379Z Generating XML reports... 2022-11-23T02:06:51.9885879Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014939.xml 2022-11-23T02:06:51.9886205Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9886585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9886737Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9887127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9887311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9887317Z 2022-11-23T02:06:51.9887414Z Running tests... 2022-11-23T02:06:51.9887683Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9888055Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16684 2022-11-23T02:06:51.9888271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16685 2022-11-23T02:06:51.9888529Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9888911Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9889079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9889466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9889648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9889883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9890269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9890437Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9890957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9891141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9891373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9891778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9892179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9892401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9892635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9892904Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9893186Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9893280Z ok (5.812s) 2022-11-23T02:06:51.9893286Z 2022-11-23T02:06:51.9893555Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9893656Z Ran 1 test in 5.812s 2022-11-23T02:06:51.9893661Z 2022-11-23T02:06:51.9893746Z OK 2022-11-23T02:06:51.9893751Z 2022-11-23T02:06:51.9893867Z Generating XML reports... 2022-11-23T02:06:51.9894314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014948.xml 2022-11-23T02:06:51.9894629Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9895074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9895250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9895649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9895831Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9895837Z 2022-11-23T02:06:51.9895940Z Running tests... 2022-11-23T02:06:51.9896206Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9896531Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16898 2022-11-23T02:06:51.9896739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16899 2022-11-23T02:06:51.9896993Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9897426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9897586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9897973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9898150Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9898367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9898740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9898900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9899286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9899469Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9899697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9900095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9900490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9900706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9900918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9901065Z skip: Skipped due to small world size. (4.810s) 2022-11-23T02:06:51.9901071Z 2022-11-23T02:06:51.9901339Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9901439Z Ran 1 test in 4.811s 2022-11-23T02:06:51.9901447Z 2022-11-23T02:06:51.9901538Z OK (skipped=1) 2022-11-23T02:06:51.9901544Z 2022-11-23T02:06:51.9901654Z Generating XML reports... 2022-11-23T02:06:51.9902098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014958.xml 2022-11-23T02:06:51.9902417Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9902792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9902955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9903339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9903519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9903525Z 2022-11-23T02:06:51.9903619Z Running tests... 2022-11-23T02:06:51.9903927Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9904245Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17101 2022-11-23T02:06:51.9904451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17102 2022-11-23T02:06:51.9904709Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9905088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9905249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9905637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9905812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9906091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9906466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9906626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9907008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9907183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9907407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9907809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9908203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9908424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9908637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9908722Z ok (5.344s) 2022-11-23T02:06:51.9908728Z 2022-11-23T02:06:51.9908991Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9909087Z Ran 1 test in 5.345s 2022-11-23T02:06:51.9909092Z 2022-11-23T02:06:51.9909168Z OK 2022-11-23T02:06:51.9909173Z 2022-11-23T02:06:51.9909284Z Generating XML reports... 2022-11-23T02:06:51.9909715Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015007.xml 2022-11-23T02:06:51.9910028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9910404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9910575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9910959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9911135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9911141Z 2022-11-23T02:06:51.9911234Z Running tests... 2022-11-23T02:06:51.9911496Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9911814Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17304 2022-11-23T02:06:51.9912019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17305 2022-11-23T02:06:51.9912272Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9912703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9912869Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9913261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9913439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9913664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9914041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9914207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9914588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9914812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9915035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9915446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9915842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9916050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9916263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9916348Z ok (5.344s) 2022-11-23T02:06:51.9916354Z 2022-11-23T02:06:51.9916618Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9916713Z Ran 1 test in 5.344s 2022-11-23T02:06:51.9916719Z 2022-11-23T02:06:51.9916800Z OK 2022-11-23T02:06:51.9916806Z 2022-11-23T02:06:51.9916915Z Generating XML reports... 2022-11-23T02:06:51.9917357Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015017.xml 2022-11-23T02:06:51.9917672Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9918045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9918212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9918595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9918773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9918779Z 2022-11-23T02:06:51.9918871Z Running tests... 2022-11-23T02:06:51.9919135Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9919472Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17507 2022-11-23T02:06:51.9919680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17508 2022-11-23T02:06:51.9919932Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9920307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9920470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9920853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9921028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9921243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9921672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9921840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9922229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9922405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9922636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9923032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9923428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9923649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9923926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9924206Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9924480Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9924566Z ok (6.122s) 2022-11-23T02:06:51.9924572Z 2022-11-23T02:06:51.9924835Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9924931Z Ran 1 test in 6.123s 2022-11-23T02:06:51.9924937Z 2022-11-23T02:06:51.9925014Z OK 2022-11-23T02:06:51.9925020Z 2022-11-23T02:06:51.9925127Z Generating XML reports... 2022-11-23T02:06:51.9925570Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015026.xml 2022-11-23T02:06:51.9925883Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9926269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9926429Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9926814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9926985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9927000Z 2022-11-23T02:06:51.9927083Z Running tests... 2022-11-23T02:06:51.9927345Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9927664Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17717 2022-11-23T02:06:51.9927871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17718 2022-11-23T02:06:51.9928131Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9928507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9928667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9929049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9929225Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9929448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9929826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9929993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9930436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9930705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9930938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9931352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9931750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9931972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9932193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9932478Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9932757Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:51.9932893Z ok (5.565s) 2022-11-23T02:06:51.9932914Z 2022-11-23T02:06:51.9933172Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9933274Z Ran 1 test in 5.565s 2022-11-23T02:06:51.9933280Z 2022-11-23T02:06:51.9933362Z OK 2022-11-23T02:06:51.9933367Z 2022-11-23T02:06:51.9933480Z Generating XML reports... 2022-11-23T02:06:51.9933924Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015036.xml 2022-11-23T02:06:51.9934241Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9934620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9934791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9935184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9935372Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9935378Z 2022-11-23T02:06:51.9935479Z Running tests... 2022-11-23T02:06:51.9935750Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9936071Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17927 2022-11-23T02:06:51.9936281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17928 2022-11-23T02:06:51.9936541Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9936921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9937093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9937495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9937675Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9937904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9938279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9938431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9938816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9938997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9939222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9939678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9940089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:51.9940444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:51.9940677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:51.9940985Z ok (5.008s) 2022-11-23T02:06:51.9940991Z 2022-11-23T02:06:51.9941284Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9941396Z Ran 1 test in 5.009s 2022-11-23T02:06:51.9941402Z 2022-11-23T02:06:51.9941492Z OK 2022-11-23T02:06:51.9941497Z 2022-11-23T02:06:51.9941640Z Generating XML reports... 2022-11-23T02:06:51.9942100Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015046.xml 2022-11-23T02:06:51.9942504Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9942899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9943111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9943514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9943682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9943710Z 2022-11-23T02:06:51.9943794Z Running tests... 2022-11-23T02:06:51.9944093Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9944349Z test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:51.9944355Z 2022-11-23T02:06:51.9944634Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9944751Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9944757Z 2022-11-23T02:06:51.9944862Z OK (skipped=1) 2022-11-23T02:06:51.9944868Z 2022-11-23T02:06:51.9945010Z Generating XML reports... 2022-11-23T02:06:51.9945472Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015055.xml 2022-11-23T02:06:51.9945812Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9946272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9946450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9946851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9947044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9947055Z 2022-11-23T02:06:51.9947160Z Running tests... 2022-11-23T02:06:51.9947461Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9948659Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.626s) 2022-11-23T02:06:51.9948676Z 2022-11-23T02:06:51.9949308Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9949522Z Ran 1 test in 0.626s 2022-11-23T02:06:51.9949544Z 2022-11-23T02:06:51.9949705Z OK (skipped=1) 2022-11-23T02:06:51.9949718Z 2022-11-23T02:06:51.9949855Z Generating XML reports... 2022-11-23T02:06:51.9950321Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015059.xml 2022-11-23T02:06:51.9950742Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9951186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9951378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9951759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9951951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9951979Z 2022-11-23T02:06:51.9952063Z Running tests... 2022-11-23T02:06:51.9952347Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9952617Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:51.9952623Z 2022-11-23T02:06:51.9952897Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9953064Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9953069Z 2022-11-23T02:06:51.9953210Z OK (skipped=1) 2022-11-23T02:06:51.9953216Z 2022-11-23T02:06:51.9953338Z Generating XML reports... 2022-11-23T02:06:51.9953808Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015104.xml 2022-11-23T02:06:51.9954139Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9954528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9954701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9955105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9955308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9955320Z 2022-11-23T02:06:51.9955452Z Running tests... 2022-11-23T02:06:51.9955732Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9956065Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:51.9956071Z 2022-11-23T02:06:51.9956348Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9956456Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9956461Z 2022-11-23T02:06:51.9956564Z OK (skipped=1) 2022-11-23T02:06:51.9956570Z 2022-11-23T02:06:51.9956695Z Generating XML reports... 2022-11-23T02:06:51.9957164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015108.xml 2022-11-23T02:06:51.9957470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9957883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9958067Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9958473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9958660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9958666Z 2022-11-23T02:06:51.9958775Z Running tests... 2022-11-23T02:06:51.9959054Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9959334Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s) 2022-11-23T02:06:51.9959339Z 2022-11-23T02:06:51.9959621Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9959756Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9959762Z 2022-11-23T02:06:51.9959868Z OK (skipped=1) 2022-11-23T02:06:51.9959874Z 2022-11-23T02:06:51.9959999Z Generating XML reports... 2022-11-23T02:06:51.9960523Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015112.xml 2022-11-23T02:06:51.9960863Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9961264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9961441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9961842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9962058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9962063Z 2022-11-23T02:06:51.9962172Z Running tests... 2022-11-23T02:06:51.9962454Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9963395Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.579s) 2022-11-23T02:06:51.9963457Z 2022-11-23T02:06:51.9963756Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9963864Z Ran 1 test in 0.579s 2022-11-23T02:06:51.9963870Z 2022-11-23T02:06:51.9964101Z OK (skipped=1) 2022-11-23T02:06:51.9964106Z 2022-11-23T02:06:51.9964294Z Generating XML reports... 2022-11-23T02:06:51.9964792Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015116.xml 2022-11-23T02:06:51.9965103Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9965497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9965681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9966096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9966286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9966292Z 2022-11-23T02:06:51.9966395Z Running tests... 2022-11-23T02:06:51.9966672Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9967206Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:06:51.9967213Z 2022-11-23T02:06:51.9967517Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9967627Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9967636Z 2022-11-23T02:06:51.9967757Z OK (skipped=1) 2022-11-23T02:06:51.9967765Z 2022-11-23T02:06:51.9967886Z Generating XML reports... 2022-11-23T02:06:51.9968350Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015121.xml 2022-11-23T02:06:51.9968681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9969070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9969278Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9969698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9969888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9969894Z 2022-11-23T02:06:51.9970010Z Running tests... 2022-11-23T02:06:51.9970292Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9971064Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:06:51.9971074Z 2022-11-23T02:06:51.9971369Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9971478Z Ran 1 test in 0.002s 2022-11-23T02:06:51.9971483Z 2022-11-23T02:06:51.9971589Z OK (skipped=1) 2022-11-23T02:06:51.9971595Z 2022-11-23T02:06:51.9971699Z Generating XML reports... 2022-11-23T02:06:51.9972192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015125.xml 2022-11-23T02:06:51.9972535Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:51.9972923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9973163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9973568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9973759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9973765Z 2022-11-23T02:06:51.9973934Z Running tests... 2022-11-23T02:06:51.9974215Z ---------------------------------------------------------------------- 2022-11-23T02:06:51.9974580Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18658 2022-11-23T02:06:51.9974801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18659 2022-11-23T02:06:51.9975066Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:51.9975465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9975645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9976047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9976243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9976498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:51.9976908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:51.9977082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:51.9977484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:51.9977671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:51.9977919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:51.9978313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0036023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0036371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0036594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0036849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpad3pf1dl 2022-11-23T02:06:52.0037101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpad3pf1dl/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0037569Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0038195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2o16s1zq 2022-11-23T02:06:52.0038451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2o16s1zq/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0038739Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0038961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0039183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0039276Z ok (7.417s) 2022-11-23T02:06:52.0039284Z 2022-11-23T02:06:52.0039566Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0039667Z Ran 1 test in 7.417s 2022-11-23T02:06:52.0039673Z 2022-11-23T02:06:52.0039764Z OK 2022-11-23T02:06:52.0039771Z 2022-11-23T02:06:52.0039896Z Generating XML reports... 2022-11-23T02:06:52.0040353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015129.xml 2022-11-23T02:06:52.0040768Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0041158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0041328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0041722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0041903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0041909Z 2022-11-23T02:06:52.0042003Z Running tests... 2022-11-23T02:06:52.0042271Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0042591Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18878 2022-11-23T02:06:52.0042804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18879 2022-11-23T02:06:52.0043056Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0043438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0043616Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0044012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0044200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0044423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0044804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0044981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0045372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0045552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0045784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0046189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0046588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0046794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0047006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0047306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxk36loj 2022-11-23T02:06:52.0047564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxk36loj/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0047802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwbtcind 2022-11-23T02:06:52.0048057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwbtcind/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0048342Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0048622Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0048847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0049074Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0049300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0049616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0049706Z ok (7.516s) 2022-11-23T02:06:52.0049712Z 2022-11-23T02:06:52.0049991Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0050092Z Ran 1 test in 7.516s 2022-11-23T02:06:52.0050099Z 2022-11-23T02:06:52.0050180Z OK 2022-11-23T02:06:52.0050185Z 2022-11-23T02:06:52.0050299Z Generating XML reports... 2022-11-23T02:06:52.0051042Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015141.xml 2022-11-23T02:06:52.0051367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0051763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0051932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0052324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0052507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0052513Z 2022-11-23T02:06:52.0052608Z Running tests... 2022-11-23T02:06:52.0052882Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0053825Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.614s) 2022-11-23T02:06:52.0053832Z 2022-11-23T02:06:52.0054109Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0054216Z Ran 1 test in 0.614s 2022-11-23T02:06:52.0054222Z 2022-11-23T02:06:52.0054319Z OK (skipped=1) 2022-11-23T02:06:52.0054325Z 2022-11-23T02:06:52.0054439Z Generating XML reports... 2022-11-23T02:06:52.0054896Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015152.xml 2022-11-23T02:06:52.0055219Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0055599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0055774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0056169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0056352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0056358Z 2022-11-23T02:06:52.0056457Z Running tests... 2022-11-23T02:06:52.0056802Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0057773Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.641s) 2022-11-23T02:06:52.0057780Z 2022-11-23T02:06:52.0058049Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0058152Z Ran 1 test in 0.641s 2022-11-23T02:06:52.0058158Z 2022-11-23T02:06:52.0058253Z OK (skipped=1) 2022-11-23T02:06:52.0058259Z 2022-11-23T02:06:52.0058377Z Generating XML reports... 2022-11-23T02:06:52.0058831Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015157.xml 2022-11-23T02:06:52.0059222Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0059604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0059775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0060165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0060344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0060350Z 2022-11-23T02:06:52.0060449Z Running tests... 2022-11-23T02:06:52.0060707Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0061192Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.003s) 2022-11-23T02:06:52.0061199Z 2022-11-23T02:06:52.0061473Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0061574Z Ran 1 test in 0.004s 2022-11-23T02:06:52.0061580Z 2022-11-23T02:06:52.0061675Z OK (skipped=1) 2022-11-23T02:06:52.0061682Z 2022-11-23T02:06:52.0061794Z Generating XML reports... 2022-11-23T02:06:52.0062245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015202.xml 2022-11-23T02:06:52.0062567Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0062946Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0063113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0063504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0063683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0063691Z 2022-11-23T02:06:52.0063791Z Running tests... 2022-11-23T02:06:52.0064060Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0064416Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19296 2022-11-23T02:06:52.0064629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19297 2022-11-23T02:06:52.0064892Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0065275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0065442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0065837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0066076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0066310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0066699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0066852Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0067243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0067424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0067654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0068056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0068515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0068732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0068949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0069185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21q25mjw 2022-11-23T02:06:52.0069427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21q25mjw/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0069701Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0069932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszl3uknu 2022-11-23T02:06:52.0070176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszl3uknu/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0070458Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0070542Z ok (5.349s) 2022-11-23T02:06:52.0070548Z 2022-11-23T02:06:52.0070811Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0070912Z Ran 1 test in 5.349s 2022-11-23T02:06:52.0070918Z 2022-11-23T02:06:52.0070993Z OK 2022-11-23T02:06:52.0070999Z 2022-11-23T02:06:52.0071103Z Generating XML reports... 2022-11-23T02:06:52.0071552Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015206.xml 2022-11-23T02:06:52.0071869Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0072247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0072399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0072791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0072972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0072977Z 2022-11-23T02:06:52.0073069Z Running tests... 2022-11-23T02:06:52.0073334Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0073644Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19506 2022-11-23T02:06:52.0073851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19507 2022-11-23T02:06:52.0074105Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0074479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0074643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0075081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0075260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0075485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0075862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0076025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0076418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0076592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0076816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0077266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0077667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0077882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0078091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0078324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2innwx42 2022-11-23T02:06:52.0078561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2innwx42/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0078836Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0079070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4klqt9fe 2022-11-23T02:06:52.0079321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4klqt9fe/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0079603Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0079820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0080038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0080129Z ok (7.521s) 2022-11-23T02:06:52.0080135Z 2022-11-23T02:06:52.0080407Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0080509Z Ran 1 test in 7.522s 2022-11-23T02:06:52.0080515Z 2022-11-23T02:06:52.0080594Z OK 2022-11-23T02:06:52.0080600Z 2022-11-23T02:06:52.0080712Z Generating XML reports... 2022-11-23T02:06:52.0081170Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015215.xml 2022-11-23T02:06:52.0081511Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0081899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0082058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0082446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0082622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0082628Z 2022-11-23T02:06:52.0082724Z Running tests... 2022-11-23T02:06:52.0082996Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0083462Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.005s) 2022-11-23T02:06:52.0083478Z 2022-11-23T02:06:52.0083736Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0083880Z Ran 1 test in 0.005s 2022-11-23T02:06:52.0083887Z 2022-11-23T02:06:52.0083980Z OK (skipped=1) 2022-11-23T02:06:52.0083986Z 2022-11-23T02:06:52.0084095Z Generating XML reports... 2022-11-23T02:06:52.0084539Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015227.xml 2022-11-23T02:06:52.0084850Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0085225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0085384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0085767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0085940Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0085995Z 2022-11-23T02:06:52.0086089Z Running tests... 2022-11-23T02:06:52.0086355Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0086823Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.004s) 2022-11-23T02:06:52.0086829Z 2022-11-23T02:06:52.0087092Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0087183Z Ran 1 test in 0.004s 2022-11-23T02:06:52.0087189Z 2022-11-23T02:06:52.0087277Z OK (skipped=1) 2022-11-23T02:06:52.0087283Z 2022-11-23T02:06:52.0087388Z Generating XML reports... 2022-11-23T02:06:52.0087833Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015231.xml 2022-11-23T02:06:52.0088147Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0088524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0088688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0089070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0089245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0089251Z 2022-11-23T02:06:52.0089335Z Running tests... 2022-11-23T02:06:52.0089596Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0089927Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s) 2022-11-23T02:06:52.0089933Z 2022-11-23T02:06:52.0090194Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0090286Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0090292Z 2022-11-23T02:06:52.0090384Z OK (skipped=1) 2022-11-23T02:06:52.0090394Z 2022-11-23T02:06:52.0090501Z Generating XML reports... 2022-11-23T02:06:52.0091036Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015235.xml 2022-11-23T02:06:52.0091350Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0091724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0091885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0092277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0092455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0092461Z 2022-11-23T02:06:52.0092561Z Running tests... 2022-11-23T02:06:52.0092832Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0093334Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.005s) 2022-11-23T02:06:52.0093341Z 2022-11-23T02:06:52.0093615Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0093710Z Ran 1 test in 0.005s 2022-11-23T02:06:52.0093715Z 2022-11-23T02:06:52.0093803Z OK (skipped=1) 2022-11-23T02:06:52.0093809Z 2022-11-23T02:06:52.0093913Z Generating XML reports... 2022-11-23T02:06:52.0094354Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015239.xml 2022-11-23T02:06:52.0094669Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0095049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0095202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0095591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0095820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0095825Z 2022-11-23T02:06:52.0095916Z Running tests... 2022-11-23T02:06:52.0096188Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0096634Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s) 2022-11-23T02:06:52.0096641Z 2022-11-23T02:06:52.0096900Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0096990Z Ran 1 test in 0.003s 2022-11-23T02:06:52.0096996Z 2022-11-23T02:06:52.0097085Z OK (skipped=1) 2022-11-23T02:06:52.0097091Z 2022-11-23T02:06:52.0097197Z Generating XML reports... 2022-11-23T02:06:52.0097641Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015243.xml 2022-11-23T02:06:52.0097965Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0098342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0098505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0098887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0099060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0099066Z 2022-11-23T02:06:52.0099156Z Running tests... 2022-11-23T02:06:52.0099436Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0100381Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.590s) 2022-11-23T02:06:52.0100391Z 2022-11-23T02:06:52.0100659Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0100753Z Ran 1 test in 0.590s 2022-11-23T02:06:52.0100758Z 2022-11-23T02:06:52.0100851Z OK (skipped=1) 2022-11-23T02:06:52.0100857Z 2022-11-23T02:06:52.0100955Z Generating XML reports... 2022-11-23T02:06:52.0101398Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015247.xml 2022-11-23T02:06:52.0101718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0102094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0102256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0102700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0102880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0102885Z 2022-11-23T02:06:52.0102981Z Running tests... 2022-11-23T02:06:52.0103252Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0104187Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.576s) 2022-11-23T02:06:52.0104194Z 2022-11-23T02:06:52.0104461Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0104610Z Ran 1 test in 0.576s 2022-11-23T02:06:52.0104616Z 2022-11-23T02:06:52.0104713Z OK (skipped=1) 2022-11-23T02:06:52.0104718Z 2022-11-23T02:06:52.0104843Z Generating XML reports... 2022-11-23T02:06:52.0105298Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015252.xml 2022-11-23T02:06:52.0105624Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0106013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0106185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0106587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0106773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0106779Z 2022-11-23T02:06:52.0106883Z Running tests... 2022-11-23T02:06:52.0107174Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0107517Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20188 2022-11-23T02:06:52.0107740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20189 2022-11-23T02:06:52.0108011Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0108394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0108547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0108948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0109136Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0109378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0109766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0109939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0110336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0110520Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0110760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0111172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0111578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0111854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0112088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:06:52.0112313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0112545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:06:52.0112962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0113373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0113615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rjk8ubt 2022-11-23T02:06:52.0113878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rjk8ubt/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0114172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7n8a2ak 2022-11-23T02:06:52.0114431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7n8a2ak/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0114720Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0115004Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0115216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0115445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0115675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0115905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0116001Z ok (7.864s) 2022-11-23T02:06:52.0116007Z 2022-11-23T02:06:52.0116281Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0116373Z Ran 1 test in 7.864s 2022-11-23T02:06:52.0116379Z 2022-11-23T02:06:52.0116451Z OK 2022-11-23T02:06:52.0116457Z 2022-11-23T02:06:52.0116562Z Generating XML reports... 2022-11-23T02:06:52.0117008Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015256.xml 2022-11-23T02:06:52.0117322Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0117702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0117878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0118268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0118464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0118470Z 2022-11-23T02:06:52.0118570Z Running tests... 2022-11-23T02:06:52.0118844Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0119171Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20410 2022-11-23T02:06:52.0119385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20411 2022-11-23T02:06:52.0119647Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0120035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0120209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0120657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0120856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0121100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0121485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0121658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0122061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0122255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0122489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0122904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0123368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0123594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0123865Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:06:52.0124088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0124354Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:06:52.0124599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsc6xnsqm 2022-11-23T02:06:52.0124855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsc6xnsqm/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0125137Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0125387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61osdkrb 2022-11-23T02:06:52.0125641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61osdkrb/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0125920Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0126144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0126366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0126573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0126806Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0127081Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:06:52.0127354Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:06:52.0127616Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:06:52.0127876Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:06:52.0128101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0128328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0128552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0128779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0129041Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:06:52.0129353Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:06:52.0129613Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:06:52.0129868Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:06:52.0130078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0130292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0130506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0130782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0130871Z ok (8.228s) 2022-11-23T02:06:52.0130878Z 2022-11-23T02:06:52.0131224Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0131333Z Ran 1 test in 8.228s 2022-11-23T02:06:52.0131339Z 2022-11-23T02:06:52.0131431Z OK 2022-11-23T02:06:52.0131436Z 2022-11-23T02:06:52.0131559Z Generating XML reports... 2022-11-23T02:06:52.0131996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015308.xml 2022-11-23T02:06:52.0132324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0132711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0132887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0133281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0133464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0133474Z 2022-11-23T02:06:52.0133573Z Running tests... 2022-11-23T02:06:52.0133854Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0134789Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.578s) 2022-11-23T02:06:52.0134796Z 2022-11-23T02:06:52.0135074Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0135177Z Ran 1 test in 0.578s 2022-11-23T02:06:52.0135183Z 2022-11-23T02:06:52.0135277Z OK (skipped=1) 2022-11-23T02:06:52.0135282Z 2022-11-23T02:06:52.0135396Z Generating XML reports... 2022-11-23T02:06:52.0135856Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015321.xml 2022-11-23T02:06:52.0136192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0136578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0136741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0137140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0137330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0137337Z 2022-11-23T02:06:52.0137442Z Running tests... 2022-11-23T02:06:52.0137716Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0138046Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20696 2022-11-23T02:06:52.0138324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20697 2022-11-23T02:06:52.0138587Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0138973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0139126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0139517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0139703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0139936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0140322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0140542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0140939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0141123Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0141365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0141771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0142179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0142405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0142953Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:06:52.0143185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0143726Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:06:52.0143970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsm1m7y0l 2022-11-23T02:06:52.0144220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsm1m7y0l/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0144505Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0144758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkosfnfz0 2022-11-23T02:06:52.0145021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkosfnfz0/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0145306Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0145529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0145754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0146025Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:06:52.0146295Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:06:52.0146579Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:06:52.0146909Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:06:52.0147238Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:06:52.0147564Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:06:52.0147878Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:06:52.0148195Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:06:52.0148404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0148671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0148771Z ok (7.514s) 2022-11-23T02:06:52.0148778Z 2022-11-23T02:06:52.0149055Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0149167Z Ran 1 test in 7.515s 2022-11-23T02:06:52.0149173Z 2022-11-23T02:06:52.0149257Z OK 2022-11-23T02:06:52.0149263Z 2022-11-23T02:06:52.0149373Z Generating XML reports... 2022-11-23T02:06:52.0149827Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015325.xml 2022-11-23T02:06:52.0150150Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0150542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0150711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0151107Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0151288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0151294Z 2022-11-23T02:06:52.0151393Z Running tests... 2022-11-23T02:06:52.0151669Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0152049Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0152055Z 2022-11-23T02:06:52.0152326Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0152430Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0152435Z 2022-11-23T02:06:52.0152525Z OK (skipped=1) 2022-11-23T02:06:52.0152530Z 2022-11-23T02:06:52.0152637Z Generating XML reports... 2022-11-23T02:06:52.0153089Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015337.xml 2022-11-23T02:06:52.0153412Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0153788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0153940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0154324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0154509Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0154514Z 2022-11-23T02:06:52.0154613Z Running tests... 2022-11-23T02:06:52.0154881Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0155312Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0155323Z 2022-11-23T02:06:52.0155601Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0155700Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0155705Z 2022-11-23T02:06:52.0155796Z OK (skipped=1) 2022-11-23T02:06:52.0155801Z 2022-11-23T02:06:52.0155920Z Generating XML reports... 2022-11-23T02:06:52.0156367Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015341.xml 2022-11-23T02:06:52.0156680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0157062Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0157228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0157626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0157852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0157857Z 2022-11-23T02:06:52.0157948Z Running tests... 2022-11-23T02:06:52.0158227Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0158668Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0158674Z 2022-11-23T02:06:52.0158938Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0159045Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0159051Z 2022-11-23T02:06:52.0159147Z OK (skipped=1) 2022-11-23T02:06:52.0159152Z 2022-11-23T02:06:52.0159266Z Generating XML reports... 2022-11-23T02:06:52.0159725Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015345.xml 2022-11-23T02:06:52.0160048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0160414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0160579Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0160963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0161149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0161154Z 2022-11-23T02:06:52.0161249Z Running tests... 2022-11-23T02:06:52.0161521Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0161957Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0161966Z 2022-11-23T02:06:52.0162229Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0162328Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0162334Z 2022-11-23T02:06:52.0162429Z OK (skipped=1) 2022-11-23T02:06:52.0162434Z 2022-11-23T02:06:52.0162545Z Generating XML reports... 2022-11-23T02:06:52.0162993Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015349.xml 2022-11-23T02:06:52.0163319Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0163698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0163866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0164310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0164488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0164494Z 2022-11-23T02:06:52.0164595Z Running tests... 2022-11-23T02:06:52.0164863Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0165294Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0165300Z 2022-11-23T02:06:52.0165570Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0165669Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0165675Z 2022-11-23T02:06:52.0165777Z OK (skipped=1) 2022-11-23T02:06:52.0165783Z 2022-11-23T02:06:52.0165940Z Generating XML reports... 2022-11-23T02:06:52.0166396Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015353.xml 2022-11-23T02:06:52.0166718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0167083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0167255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0167647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0167821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0167827Z 2022-11-23T02:06:52.0167929Z Running tests... 2022-11-23T02:06:52.0168202Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0168635Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0168646Z 2022-11-23T02:06:52.0168917Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0169019Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0169025Z 2022-11-23T02:06:52.0169114Z OK (skipped=1) 2022-11-23T02:06:52.0169119Z 2022-11-23T02:06:52.0169238Z Generating XML reports... 2022-11-23T02:06:52.0169680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015357.xml 2022-11-23T02:06:52.0170004Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0170385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0170696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0171103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0171287Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0171293Z 2022-11-23T02:06:52.0171390Z Running tests... 2022-11-23T02:06:52.0171663Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0172096Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0172103Z 2022-11-23T02:06:52.0172364Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0172473Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0172479Z 2022-11-23T02:06:52.0172576Z OK (skipped=1) 2022-11-23T02:06:52.0172585Z 2022-11-23T02:06:52.0172772Z Generating XML reports... 2022-11-23T02:06:52.0173234Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015401.xml 2022-11-23T02:06:52.0173540Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0173915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0174086Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0174475Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0174653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0174659Z 2022-11-23T02:06:52.0174758Z Running tests... 2022-11-23T02:06:52.0175021Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0175523Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0175530Z 2022-11-23T02:06:52.0175798Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0175899Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0175905Z 2022-11-23T02:06:52.0176004Z OK (skipped=1) 2022-11-23T02:06:52.0176010Z 2022-11-23T02:06:52.0176120Z Generating XML reports... 2022-11-23T02:06:52.0176568Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015405.xml 2022-11-23T02:06:52.0176888Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0177265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0177433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0177816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0177991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0177997Z 2022-11-23T02:06:52.0178091Z Running tests... 2022-11-23T02:06:52.0178354Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0178778Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0178785Z 2022-11-23T02:06:52.0179047Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0179141Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0179147Z 2022-11-23T02:06:52.0179245Z OK (skipped=1) 2022-11-23T02:06:52.0179250Z 2022-11-23T02:06:52.0179362Z Generating XML reports... 2022-11-23T02:06:52.0179802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015409.xml 2022-11-23T02:06:52.0180107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0180479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0180639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0181021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0181194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0181199Z 2022-11-23T02:06:52.0181289Z Running tests... 2022-11-23T02:06:52.0181552Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0182110Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0182117Z 2022-11-23T02:06:52.0182386Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0182480Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0182485Z 2022-11-23T02:06:52.0182575Z OK (skipped=1) 2022-11-23T02:06:52.0182581Z 2022-11-23T02:06:52.0182688Z Generating XML reports... 2022-11-23T02:06:52.0183127Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015413.xml 2022-11-23T02:06:52.0183440Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0183814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0184037Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0184422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0184596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0184602Z 2022-11-23T02:06:52.0184693Z Running tests... 2022-11-23T02:06:52.0184955Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0185327Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0185334Z 2022-11-23T02:06:52.0185597Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0185691Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0185697Z 2022-11-23T02:06:52.0185792Z OK (skipped=1) 2022-11-23T02:06:52.0185798Z 2022-11-23T02:06:52.0185909Z Generating XML reports... 2022-11-23T02:06:52.0186342Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015417.xml 2022-11-23T02:06:52.0186656Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0187028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0187189Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0187575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0187750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0187755Z 2022-11-23T02:06:52.0187847Z Running tests... 2022-11-23T02:06:52.0188109Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0188489Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:06:52.0188495Z 2022-11-23T02:06:52.0188757Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0188851Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0188856Z 2022-11-23T02:06:52.0188946Z OK (skipped=1) 2022-11-23T02:06:52.0188951Z 2022-11-23T02:06:52.0189058Z Generating XML reports... 2022-11-23T02:06:52.0189499Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015421.xml 2022-11-23T02:06:52.0189811Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0190183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0190396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0190785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0190960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0190965Z 2022-11-23T02:06:52.0191060Z Running tests... 2022-11-23T02:06:52.0191322Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0191758Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s) 2022-11-23T02:06:52.0191765Z 2022-11-23T02:06:52.0192023Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0192118Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0192124Z 2022-11-23T02:06:52.0192206Z OK (skipped=1) 2022-11-23T02:06:52.0192220Z 2022-11-23T02:06:52.0192319Z Generating XML reports... 2022-11-23T02:06:52.0192814Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015425.xml 2022-11-23T02:06:52.0193127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0193501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0193659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0194042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0194218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0194224Z 2022-11-23T02:06:52.0194322Z Running tests... 2022-11-23T02:06:52.0194583Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0194886Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21774 2022-11-23T02:06:52.0195095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21775 2022-11-23T02:06:52.0195353Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0195724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0195884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0196267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0196441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0196665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0197040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0197205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0197588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0197761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0197977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0198377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0198777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0198994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0199258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0199498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxoyogcg 2022-11-23T02:06:52.0199746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxoyogcg/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0200026Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0200259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7faopap 2022-11-23T02:06:52.0200507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7faopap/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0200783Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0200867Z ok (7.615s) 2022-11-23T02:06:52.0200873Z 2022-11-23T02:06:52.0201135Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0201230Z Ran 1 test in 7.615s 2022-11-23T02:06:52.0201331Z 2022-11-23T02:06:52.0201407Z OK 2022-11-23T02:06:52.0201417Z 2022-11-23T02:06:52.0201525Z Generating XML reports... 2022-11-23T02:06:52.0201967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015429.xml 2022-11-23T02:06:52.0202280Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0202653Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0202814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0203197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0203371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0203377Z 2022-11-23T02:06:52.0203461Z Running tests... 2022-11-23T02:06:52.0203723Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0204049Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21987 2022-11-23T02:06:52.0204257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21988 2022-11-23T02:06:52.0204510Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0204884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0205045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0205427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0205601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0205828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0206204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0206364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0206747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0206921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0207145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0207545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0207946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0208215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0208430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0208663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6qvu0uf 2022-11-23T02:06:52.0208909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6qvu0uf/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0209144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa52hporr 2022-11-23T02:06:52.0209388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa52hporr/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0209663Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0209937Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0210154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0210422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0210881Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:06:52.0211029Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:06:52.0211424Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:06:52.0211570Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:06:52.0211655Z ok (7.546s) 2022-11-23T02:06:52.0211661Z 2022-11-23T02:06:52.0211923Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0212017Z Ran 1 test in 7.546s 2022-11-23T02:06:52.0212022Z 2022-11-23T02:06:52.0212097Z OK 2022-11-23T02:06:52.0212103Z 2022-11-23T02:06:52.0212209Z Generating XML reports... 2022-11-23T02:06:52.0212657Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015441.xml 2022-11-23T02:06:52.0212975Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0213347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0213507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0213893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0214068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0214074Z 2022-11-23T02:06:52.0214164Z Running tests... 2022-11-23T02:06:52.0214417Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0214677Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.007s) 2022-11-23T02:06:52.0214694Z 2022-11-23T02:06:52.0214950Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0215045Z Ran 1 test in 0.007s 2022-11-23T02:06:52.0215051Z 2022-11-23T02:06:52.0215140Z OK (skipped=1) 2022-11-23T02:06:52.0215146Z 2022-11-23T02:06:52.0215252Z Generating XML reports... 2022-11-23T02:06:52.0215691Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015453.xml 2022-11-23T02:06:52.0216003Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0216376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0216536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0216918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0217160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0217167Z 2022-11-23T02:06:52.0217258Z Running tests... 2022-11-23T02:06:52.0217526Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0217833Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22273 2022-11-23T02:06:52.0218037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22274 2022-11-23T02:06:52.0218291Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0218663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0218823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0219205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0219434Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0219659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0220036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0220187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0220571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0220747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0220972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0221375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0221779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0221994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0222207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0222442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2z_ojr7h 2022-11-23T02:06:52.0222687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2z_ojr7h/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0222920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp704regdb 2022-11-23T02:06:52.0223164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp704regdb/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0223439Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0223719Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0223936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0224152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0224237Z ok (7.828s) 2022-11-23T02:06:52.0224244Z 2022-11-23T02:06:52.0224508Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0224603Z Ran 1 test in 7.829s 2022-11-23T02:06:52.0224608Z 2022-11-23T02:06:52.0224683Z OK 2022-11-23T02:06:52.0224689Z 2022-11-23T02:06:52.0224797Z Generating XML reports... 2022-11-23T02:06:52.0225230Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015457.xml 2022-11-23T02:06:52.0225544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0225975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0226140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0226528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0226702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0226708Z 2022-11-23T02:06:52.0226799Z Running tests... 2022-11-23T02:06:52.0227062Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0227531Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:06:52.0227538Z 2022-11-23T02:06:52.0227797Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0227892Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0227948Z 2022-11-23T02:06:52.0228039Z OK (skipped=1) 2022-11-23T02:06:52.0228047Z 2022-11-23T02:06:52.0228155Z Generating XML reports... 2022-11-23T02:06:52.0228601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015509.xml 2022-11-23T02:06:52.0228914Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0229288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0229448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0229834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0230008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0230014Z 2022-11-23T02:06:52.0230106Z Running tests... 2022-11-23T02:06:52.0230369Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0230837Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s) 2022-11-23T02:06:52.0230843Z 2022-11-23T02:06:52.0231102Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0231199Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0231205Z 2022-11-23T02:06:52.0231288Z OK (skipped=1) 2022-11-23T02:06:52.0231301Z 2022-11-23T02:06:52.0231401Z Generating XML reports... 2022-11-23T02:06:52.0231842Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015513.xml 2022-11-23T02:06:52.0232158Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0232531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0232694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0233081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0233255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0233261Z 2022-11-23T02:06:52.0233352Z Running tests... 2022-11-23T02:06:52.0233614Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0234112Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:06:52.0234119Z 2022-11-23T02:06:52.0234380Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0234475Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0234480Z 2022-11-23T02:06:52.0234571Z OK (skipped=1) 2022-11-23T02:06:52.0234577Z 2022-11-23T02:06:52.0234684Z Generating XML reports... 2022-11-23T02:06:52.0235177Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015517.xml 2022-11-23T02:06:52.0235497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0235870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0236029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0236413Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0236590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0236596Z 2022-11-23T02:06:52.0236687Z Running tests... 2022-11-23T02:06:52.0236949Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0237419Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:06:52.0237479Z 2022-11-23T02:06:52.0237741Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0237828Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0237833Z 2022-11-23T02:06:52.0237924Z OK (skipped=1) 2022-11-23T02:06:52.0237929Z 2022-11-23T02:06:52.0238037Z Generating XML reports... 2022-11-23T02:06:52.0238478Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015521.xml 2022-11-23T02:06:52.0238791Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0239166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0239326Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0239711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0239889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0239895Z 2022-11-23T02:06:52.0239985Z Running tests... 2022-11-23T02:06:52.0240249Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0240671Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.003s) 2022-11-23T02:06:52.0240678Z 2022-11-23T02:06:52.0240941Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0241040Z Ran 1 test in 0.003s 2022-11-23T02:06:52.0241046Z 2022-11-23T02:06:52.0241145Z OK (skipped=1) 2022-11-23T02:06:52.0241150Z 2022-11-23T02:06:52.0241261Z Generating XML reports... 2022-11-23T02:06:52.0241702Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015525.xml 2022-11-23T02:06:52.0242022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0242396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0242557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0242938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0243114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0243120Z 2022-11-23T02:06:52.0243203Z Running tests... 2022-11-23T02:06:52.0243466Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0243771Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22823 2022-11-23T02:06:52.0243976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22824 2022-11-23T02:06:52.0244285Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0244664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0244826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0245209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0245382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0245606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0245977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0246137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0246573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0246747Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0246970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0247370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0247762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0247978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0248192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0248426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdediw_5e 2022-11-23T02:06:52.0248680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdediw_5e/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0248911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvwp7gpf 2022-11-23T02:06:52.0249159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvwp7gpf/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0249428Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0249701Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0250473Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:06:52.0251397Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:06:52.0251618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0251835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0251919Z ok (7.531s) 2022-11-23T02:06:52.0251930Z 2022-11-23T02:06:52.0252280Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0252371Z Ran 1 test in 7.531s 2022-11-23T02:06:52.0252384Z 2022-11-23T02:06:52.0252452Z OK 2022-11-23T02:06:52.0252458Z 2022-11-23T02:06:52.0252566Z Generating XML reports... 2022-11-23T02:06:52.0253014Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015529.xml 2022-11-23T02:06:52.0253326Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0253704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0253863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0254248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0254483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0254493Z 2022-11-23T02:06:52.0254584Z Running tests... 2022-11-23T02:06:52.0254854Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0255802Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.585s) 2022-11-23T02:06:52.0255809Z 2022-11-23T02:06:52.0256073Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0256168Z Ran 1 test in 0.586s 2022-11-23T02:06:52.0256173Z 2022-11-23T02:06:52.0256264Z OK (skipped=1) 2022-11-23T02:06:52.0256269Z 2022-11-23T02:06:52.0256379Z Generating XML reports... 2022-11-23T02:06:52.0256829Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015541.xml 2022-11-23T02:06:52.0257141Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0257513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0257673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0258057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0258233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0258239Z 2022-11-23T02:06:52.0258330Z Running tests... 2022-11-23T02:06:52.0258591Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0259051Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:06:52.0259061Z 2022-11-23T02:06:52.0259323Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0259410Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0259423Z 2022-11-23T02:06:52.0259505Z OK (skipped=1) 2022-11-23T02:06:52.0259511Z 2022-11-23T02:06:52.0259618Z Generating XML reports... 2022-11-23T02:06:52.0260057Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015546.xml 2022-11-23T02:06:52.0260368Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0260742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0260903Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0261286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0261519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0261525Z 2022-11-23T02:06:52.0261618Z Running tests... 2022-11-23T02:06:52.0261886Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0262335Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:06:52.0262341Z 2022-11-23T02:06:52.0262601Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0262695Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0262700Z 2022-11-23T02:06:52.0262790Z OK (skipped=1) 2022-11-23T02:06:52.0262796Z 2022-11-23T02:06:52.0262902Z Generating XML reports... 2022-11-23T02:06:52.0263340Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015550.xml 2022-11-23T02:06:52.0263709Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0264083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0264244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0264626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0264799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0264805Z 2022-11-23T02:06:52.0264896Z Running tests... 2022-11-23T02:06:52.0265151Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0265463Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23241 2022-11-23T02:06:52.0265668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23242 2022-11-23T02:06:52.0265935Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0266308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0266468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0266853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0267028Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0267250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0267622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0267781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0268172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0268345Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0268568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0268969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0269364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0269579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0269792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0270026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6hubuob 2022-11-23T02:06:52.0270329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6hubuob/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0270562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogbi1ski 2022-11-23T02:06:52.0270812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogbi1ski/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0271090Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0271355Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0271440Z ok (5.454s) 2022-11-23T02:06:52.0271446Z 2022-11-23T02:06:52.0271709Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0271803Z Ran 1 test in 5.455s 2022-11-23T02:06:52.0271809Z 2022-11-23T02:06:52.0271884Z OK 2022-11-23T02:06:52.0271890Z 2022-11-23T02:06:52.0271997Z Generating XML reports... 2022-11-23T02:06:52.0272490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015554.xml 2022-11-23T02:06:52.0272806Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0273180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0273339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0273723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0273897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0273903Z 2022-11-23T02:06:52.0273994Z Running tests... 2022-11-23T02:06:52.0274256Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0275195Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.584s) 2022-11-23T02:06:52.0275206Z 2022-11-23T02:06:52.0275465Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0275559Z Ran 1 test in 0.584s 2022-11-23T02:06:52.0275565Z 2022-11-23T02:06:52.0275655Z OK (skipped=1) 2022-11-23T02:06:52.0275660Z 2022-11-23T02:06:52.0275767Z Generating XML reports... 2022-11-23T02:06:52.0276207Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015603.xml 2022-11-23T02:06:52.0276523Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0276894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0277064Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0277446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0277613Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0277627Z 2022-11-23T02:06:52.0277711Z Running tests... 2022-11-23T02:06:52.0277976Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0278440Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s) 2022-11-23T02:06:52.0278446Z 2022-11-23T02:06:52.0278705Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0278800Z Ran 1 test in 0.003s 2022-11-23T02:06:52.0278806Z 2022-11-23T02:06:52.0278899Z OK (skipped=1) 2022-11-23T02:06:52.0278908Z 2022-11-23T02:06:52.0279016Z Generating XML reports... 2022-11-23T02:06:52.0279509Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015608.xml 2022-11-23T02:06:52.0279828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0280203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0280362Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0280745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0280921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0280927Z 2022-11-23T02:06:52.0281018Z Running tests... 2022-11-23T02:06:52.0281280Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0282266Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.580s) 2022-11-23T02:06:52.0282273Z 2022-11-23T02:06:52.0282531Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0282626Z Ran 1 test in 0.580s 2022-11-23T02:06:52.0282632Z 2022-11-23T02:06:52.0282722Z OK (skipped=1) 2022-11-23T02:06:52.0282727Z 2022-11-23T02:06:52.0282835Z Generating XML reports... 2022-11-23T02:06:52.0283276Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015612.xml 2022-11-23T02:06:52.0283588Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0283971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0284131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0284515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0284683Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0284696Z 2022-11-23T02:06:52.0284780Z Running tests... 2022-11-23T02:06:52.0285043Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0285355Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23649 2022-11-23T02:06:52.0285561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23650 2022-11-23T02:06:52.0285816Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0286196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0286355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0286739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0286914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0287140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0287512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0287671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0288053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0288281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0288510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0288912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0289309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0289522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0289732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0289969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvmt1x_e 2022-11-23T02:06:52.0290218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvmt1x_e/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0290500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxm2yqz4n 2022-11-23T02:06:52.0290829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxm2yqz4n/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0291105Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0291378Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0291711Z STAGE:2022-11-23 01:56:20 23650:23650 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0292040Z STAGE:2022-11-23 01:56:20 23649:23649 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0292679Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:52.0292776Z warnings.warn( 2022-11-23T02:06:52.0292997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0293637Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:06:52.0293733Z warnings.warn( 2022-11-23T02:06:52.0293949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:06:52.0294290Z STAGE:2022-11-23 01:56:20 23649:23649 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0294625Z STAGE:2022-11-23 01:56:20 23650:23650 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0294981Z STAGE:2022-11-23 01:56:20 23650:23650 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0295338Z STAGE:2022-11-23 01:56:20 23649:23649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0295560Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:06:52.0295782Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:06:52.0296118Z STAGE:2022-11-23 01:56:20 23649:23649 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0296457Z STAGE:2022-11-23 01:56:20 23649:23649 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0296809Z STAGE:2022-11-23 01:56:20 23649:23649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0296897Z ok (6.845s) 2022-11-23T02:06:52.0296904Z 2022-11-23T02:06:52.0297169Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0297265Z Ran 1 test in 6.846s 2022-11-23T02:06:52.0297271Z 2022-11-23T02:06:52.0297350Z OK 2022-11-23T02:06:52.0297356Z 2022-11-23T02:06:52.0297512Z Generating XML reports... 2022-11-23T02:06:52.0297967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015617.xml 2022-11-23T02:06:52.0298282Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0298655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0298814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0299199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0299374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0299380Z 2022-11-23T02:06:52.0299472Z Running tests... 2022-11-23T02:06:52.0299735Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0300097Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23873 2022-11-23T02:06:52.0300303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23874 2022-11-23T02:06:52.0300557Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0300932Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0301091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0301474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0301647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0301872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0302250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0302409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0302792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0302967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0303189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0303581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0303976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0304193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0304413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0304645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9jh07xm 2022-11-23T02:06:52.0304893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9jh07xm/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0305127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpck_58dg_ 2022-11-23T02:06:52.0305369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpck_58dg_/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0305646Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0305919Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0306004Z ok (5.612s) 2022-11-23T02:06:52.0306010Z 2022-11-23T02:06:52.0306272Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0306420Z Ran 1 test in 5.613s 2022-11-23T02:06:52.0306427Z 2022-11-23T02:06:52.0306505Z OK 2022-11-23T02:06:52.0306510Z 2022-11-23T02:06:52.0306618Z Generating XML reports... 2022-11-23T02:06:52.0307064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015628.xml 2022-11-23T02:06:52.0307379Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0307751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0307911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0308295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0308471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0308527Z 2022-11-23T02:06:52.0308623Z Running tests... 2022-11-23T02:06:52.0308886Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0309206Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24087 2022-11-23T02:06:52.0309413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24088 2022-11-23T02:06:52.0309669Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0310043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0310205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0310594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0310770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0311003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0311377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0311537Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0311920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0312095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0312319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0312718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0313118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0313336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0313557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0313793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1h94ibm 2022-11-23T02:06:52.0314038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1h94ibm/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0314315Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0314552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84ted0ri 2022-11-23T02:06:52.0314790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84ted0ri/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0315063Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0315151Z ok (5.409s) 2022-11-23T02:06:52.0315207Z 2022-11-23T02:06:52.0315475Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0315570Z Ran 1 test in 5.409s 2022-11-23T02:06:52.0315576Z 2022-11-23T02:06:52.0315651Z OK 2022-11-23T02:06:52.0315657Z 2022-11-23T02:06:52.0315771Z Generating XML reports... 2022-11-23T02:06:52.0316213Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015637.xml 2022-11-23T02:06:52.0316526Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0316899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0317060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0317446Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0317689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0317695Z 2022-11-23T02:06:52.0317787Z Running tests... 2022-11-23T02:06:52.0318052Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0318993Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.583s) 2022-11-23T02:06:52.0319000Z 2022-11-23T02:06:52.0319265Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0319362Z Ran 1 test in 0.583s 2022-11-23T02:06:52.0319367Z 2022-11-23T02:06:52.0319459Z OK (skipped=1) 2022-11-23T02:06:52.0319464Z 2022-11-23T02:06:52.0319572Z Generating XML reports... 2022-11-23T02:06:52.0320020Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015647.xml 2022-11-23T02:06:52.0320335Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0320708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0320868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0321252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0321418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0321432Z 2022-11-23T02:06:52.0321515Z Running tests... 2022-11-23T02:06:52.0321780Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0322691Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.582s) 2022-11-23T02:06:52.0322712Z 2022-11-23T02:06:52.0322963Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0323059Z Ran 1 test in 0.583s 2022-11-23T02:06:52.0323065Z 2022-11-23T02:06:52.0323157Z OK (skipped=1) 2022-11-23T02:06:52.0323162Z 2022-11-23T02:06:52.0323269Z Generating XML reports... 2022-11-23T02:06:52.0323710Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015652.xml 2022-11-23T02:06:52.0324026Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0324461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0324634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0325025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0325198Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0325204Z 2022-11-23T02:06:52.0325295Z Running tests... 2022-11-23T02:06:52.0325559Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0326512Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.588s) 2022-11-23T02:06:52.0326563Z 2022-11-23T02:06:52.0326833Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0326928Z Ran 1 test in 0.588s 2022-11-23T02:06:52.0326934Z 2022-11-23T02:06:52.0327025Z OK (skipped=1) 2022-11-23T02:06:52.0327031Z 2022-11-23T02:06:52.0327139Z Generating XML reports... 2022-11-23T02:06:52.0327580Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015656.xml 2022-11-23T02:06:52.0327894Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0328266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0328428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0328810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0328989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0328999Z 2022-11-23T02:06:52.0329091Z Running tests... 2022-11-23T02:06:52.0329347Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0329838Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s) 2022-11-23T02:06:52.0329853Z 2022-11-23T02:06:52.0330104Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0330201Z Ran 1 test in 0.003s 2022-11-23T02:06:52.0330206Z 2022-11-23T02:06:52.0330299Z OK (skipped=1) 2022-11-23T02:06:52.0330305Z 2022-11-23T02:06:52.0330412Z Generating XML reports... 2022-11-23T02:06:52.0330906Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015701.xml 2022-11-23T02:06:52.0331220Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0331599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0331759Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0332142Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0332318Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0332323Z 2022-11-23T02:06:52.0332414Z Running tests... 2022-11-23T02:06:52.0332675Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0332988Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24561 2022-11-23T02:06:52.0333193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24562 2022-11-23T02:06:52.0333507Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0333890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0334050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0334434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0334608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0334837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0335209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0335370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0335748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0335976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0336202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0336605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0337001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0337215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0337432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0337817Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:06:52.0338084Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:06:52.0338320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqrmo6n20 2022-11-23T02:06:52.0338569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqrmo6n20/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0338952Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:06:52.0339205Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:06:52.0339440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt42lvkok 2022-11-23T02:06:52.0339686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt42lvkok/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0339965Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0340236Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0340324Z ok (5.714s) 2022-11-23T02:06:52.0340333Z 2022-11-23T02:06:52.0340596Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0340692Z Ran 1 test in 5.715s 2022-11-23T02:06:52.0340698Z 2022-11-23T02:06:52.0340773Z OK 2022-11-23T02:06:52.0340778Z 2022-11-23T02:06:52.0340877Z Generating XML reports... 2022-11-23T02:06:52.0341320Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015705.xml 2022-11-23T02:06:52.0341635Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0342009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0342169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0342554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0342785Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0343670Z 2022-11-23T02:06:52.0343769Z Running tests... 2022-11-23T02:06:52.0344043Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0344347Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24771 2022-11-23T02:06:52.0344555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24772 2022-11-23T02:06:52.0344811Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0345184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0345343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0345733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0345985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0346216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0346593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0346756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0347146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0347320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0347544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0347945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0348347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0348554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0348777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:06:52.0348990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0349211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:06:52.0349609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0350000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0350089Z ok (5.074s) 2022-11-23T02:06:52.0350099Z 2022-11-23T02:06:52.0350362Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0350457Z Ran 1 test in 5.074s 2022-11-23T02:06:52.0350463Z 2022-11-23T02:06:52.0350539Z OK 2022-11-23T02:06:52.0350544Z 2022-11-23T02:06:52.0350652Z Generating XML reports... 2022-11-23T02:06:52.0351097Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015715.xml 2022-11-23T02:06:52.0351408Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0351783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0351945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0352329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0352557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0352564Z 2022-11-23T02:06:52.0352658Z Running tests... 2022-11-23T02:06:52.0352926Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0353224Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24976 2022-11-23T02:06:52.0353429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24977 2022-11-23T02:06:52.0353685Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0354060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0354212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0354599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0354828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0355052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0355427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0355589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0355973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0356148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0356374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0356773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0357179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0357393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0357615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:06:52.0357828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0358050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:06:52.0358447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0358838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0358927Z ok (4.907s) 2022-11-23T02:06:52.0358932Z 2022-11-23T02:06:52.0359199Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0359294Z Ran 1 test in 4.908s 2022-11-23T02:06:52.0359300Z 2022-11-23T02:06:52.0359376Z OK 2022-11-23T02:06:52.0359382Z 2022-11-23T02:06:52.0359491Z Generating XML reports... 2022-11-23T02:06:52.0359925Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015724.xml 2022-11-23T02:06:52.0360242Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0360617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0360779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0361163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0361395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0361401Z 2022-11-23T02:06:52.0361497Z Running tests... 2022-11-23T02:06:52.0361766Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0362699Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.581s) 2022-11-23T02:06:52.0362706Z 2022-11-23T02:06:52.0362968Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0363064Z Ran 1 test in 0.581s 2022-11-23T02:06:52.0363070Z 2022-11-23T02:06:52.0363161Z OK (skipped=1) 2022-11-23T02:06:52.0363167Z 2022-11-23T02:06:52.0363277Z Generating XML reports... 2022-11-23T02:06:52.0363774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015733.xml 2022-11-23T02:06:52.0364088Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0364463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0364625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0365011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0365188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0365193Z 2022-11-23T02:06:52.0365286Z Running tests... 2022-11-23T02:06:52.0365549Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0366008Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:06:52.0366017Z 2022-11-23T02:06:52.0366282Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0366378Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0366384Z 2022-11-23T02:06:52.0366475Z OK (skipped=1) 2022-11-23T02:06:52.0366481Z 2022-11-23T02:06:52.0366589Z Generating XML reports... 2022-11-23T02:06:52.0367031Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015738.xml 2022-11-23T02:06:52.0367337Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0367711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0367873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0368261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0368440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0368445Z 2022-11-23T02:06:52.0368537Z Running tests... 2022-11-23T02:06:52.0368800Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0369113Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25313 2022-11-23T02:06:52.0369318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25314 2022-11-23T02:06:52.0369576Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0369949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0370110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0370690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0370874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0371099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0371482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0371642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0372026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0372200Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0372423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0372889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0373285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0373501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0373705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0373790Z ok (5.411s) 2022-11-23T02:06:52.0373796Z 2022-11-23T02:06:52.0374059Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0374154Z Ran 1 test in 5.411s 2022-11-23T02:06:52.0374159Z 2022-11-23T02:06:52.0374236Z OK 2022-11-23T02:06:52.0374242Z 2022-11-23T02:06:52.0374350Z Generating XML reports... 2022-11-23T02:06:52.0374792Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015742.xml 2022-11-23T02:06:52.0375114Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0375487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0375647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0376033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0376208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0376213Z 2022-11-23T02:06:52.0376306Z Running tests... 2022-11-23T02:06:52.0376569Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0376805Z test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:52.0376811Z 2022-11-23T02:06:52.0377076Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0377182Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0377188Z 2022-11-23T02:06:52.0377280Z OK (skipped=1) 2022-11-23T02:06:52.0377285Z 2022-11-23T02:06:52.0377394Z Generating XML reports... 2022-11-23T02:06:52.0377839Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015751.xml 2022-11-23T02:06:52.0378152Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0378528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0378680Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0379065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0379241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0379253Z 2022-11-23T02:06:52.0379345Z Running tests... 2022-11-23T02:06:52.0379701Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0379955Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0379961Z 2022-11-23T02:06:52.0380226Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0380322Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0380328Z 2022-11-23T02:06:52.0380420Z OK (skipped=1) 2022-11-23T02:06:52.0380425Z 2022-11-23T02:06:52.0380536Z Generating XML reports... 2022-11-23T02:06:52.0380980Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015755.xml 2022-11-23T02:06:52.0381296Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0381672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0381886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0382275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0382455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0382461Z 2022-11-23T02:06:52.0382553Z Running tests... 2022-11-23T02:06:52.0382818Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0383115Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25648 2022-11-23T02:06:52.0383322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25649 2022-11-23T02:06:52.0383578Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0383961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0384113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0384499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0384676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0384902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0385277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0385441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0385831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0386007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0386233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0386635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0387031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0387249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0387469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0387809Z STAGE:2022-11-23 01:58:02 25648:25648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0388093Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0388423Z STAGE:2022-11-23 01:58:03 25649:25649 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0388768Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0389116Z STAGE:2022-11-23 01:58:03 25648:25648 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0389472Z STAGE:2022-11-23 01:58:03 25648:25648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0389698Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0390038Z STAGE:2022-11-23 01:58:03 25649:25649 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0390393Z STAGE:2022-11-23 01:58:03 25649:25649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0390620Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0390944Z STAGE:2022-11-23 01:58:03 25648:25648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0391326Z STAGE:2022-11-23 01:58:03 25649:25649 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0391664Z STAGE:2022-11-23 01:58:03 25649:25649 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0392015Z STAGE:2022-11-23 01:58:03 25649:25649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0392355Z STAGE:2022-11-23 01:58:03 25648:25648 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0392709Z STAGE:2022-11-23 01:58:03 25648:25648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0392798Z ok (6.168s) 2022-11-23T02:06:52.0392804Z 2022-11-23T02:06:52.0393069Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0393167Z Ran 1 test in 6.168s 2022-11-23T02:06:52.0393173Z 2022-11-23T02:06:52.0393249Z OK 2022-11-23T02:06:52.0393259Z 2022-11-23T02:06:52.0393366Z Generating XML reports... 2022-11-23T02:06:52.0393813Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015759.xml 2022-11-23T02:06:52.0394127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0394504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0394669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0395054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0395231Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0395236Z 2022-11-23T02:06:52.0395333Z Running tests... 2022-11-23T02:06:52.0395600Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0395857Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:52.0395863Z 2022-11-23T02:06:52.0396126Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0396226Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0396231Z 2022-11-23T02:06:52.0396314Z OK (skipped=1) 2022-11-23T02:06:52.0396331Z 2022-11-23T02:06:52.0396430Z Generating XML reports... 2022-11-23T02:06:52.0396876Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015810.xml 2022-11-23T02:06:52.0397193Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0397572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0397735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0398176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0398362Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0398368Z 2022-11-23T02:06:52.0398460Z Running tests... 2022-11-23T02:06:52.0398729Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0398981Z test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0398987Z 2022-11-23T02:06:52.0399253Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0399352Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0399358Z 2022-11-23T02:06:52.0399449Z OK (skipped=1) 2022-11-23T02:06:52.0399455Z 2022-11-23T02:06:52.0399563Z Generating XML reports... 2022-11-23T02:06:52.0400002Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015814.xml 2022-11-23T02:06:52.0400371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0400750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0400914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0401297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0401476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0401482Z 2022-11-23T02:06:52.0401582Z Running tests... 2022-11-23T02:06:52.0401838Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0402141Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25996 2022-11-23T02:06:52.0402354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25997 2022-11-23T02:06:52.0402615Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0402989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0403154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0403538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0403717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0403946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0404323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0404483Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0404870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0405046Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0405272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0405677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0406071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0406283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0406496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0406774Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0407109Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0407201Z ok (6.477s) 2022-11-23T02:06:52.0407207Z 2022-11-23T02:06:52.0407474Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0407561Z Ran 1 test in 6.478s 2022-11-23T02:06:52.0407576Z 2022-11-23T02:06:52.0407643Z OK 2022-11-23T02:06:52.0407649Z 2022-11-23T02:06:52.0407764Z Generating XML reports... 2022-11-23T02:06:52.0408207Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015818.xml 2022-11-23T02:06:52.0408522Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0408896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0409057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0409443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0409673Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0409678Z 2022-11-23T02:06:52.0409771Z Running tests... 2022-11-23T02:06:52.0410038Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0411036Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.590s) 2022-11-23T02:06:52.0411043Z 2022-11-23T02:06:52.0411308Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0411404Z Ran 1 test in 0.590s 2022-11-23T02:06:52.0411410Z 2022-11-23T02:06:52.0411507Z OK (skipped=1) 2022-11-23T02:06:52.0411512Z 2022-11-23T02:06:52.0411625Z Generating XML reports... 2022-11-23T02:06:52.0412072Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015829.xml 2022-11-23T02:06:52.0412391Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0412763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0412925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0413309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0413486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0413492Z 2022-11-23T02:06:52.0413584Z Running tests... 2022-11-23T02:06:52.0413848Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0414154Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26272 2022-11-23T02:06:52.0414361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26273 2022-11-23T02:06:52.0414608Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0414986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0415152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0415536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0415718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0415943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0416381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0416546Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0416937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0417118Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0417346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0417746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0418141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0418356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0418640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:06:52.0418856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0419080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:06:52.0419481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0419880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0419966Z ok (5.012s) 2022-11-23T02:06:52.0419972Z 2022-11-23T02:06:52.0420235Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0420332Z Ran 1 test in 5.013s 2022-11-23T02:06:52.0420338Z 2022-11-23T02:06:52.0420419Z OK 2022-11-23T02:06:52.0420425Z 2022-11-23T02:06:52.0420528Z Generating XML reports... 2022-11-23T02:06:52.0420970Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015833.xml 2022-11-23T02:06:52.0421285Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0421658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0421820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0422210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0422388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0422394Z 2022-11-23T02:06:52.0422487Z Running tests... 2022-11-23T02:06:52.0422753Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0423057Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26477 2022-11-23T02:06:52.0423266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26478 2022-11-23T02:06:52.0423521Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0423893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0424053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0424440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0424616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0424840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0425268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0425433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0425819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0425999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0426227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0426614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0427011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0427226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0427489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0427767Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0428042Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0428129Z ok (6.015s) 2022-11-23T02:06:52.0428134Z 2022-11-23T02:06:52.0428398Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0428496Z Ran 1 test in 6.016s 2022-11-23T02:06:52.0428502Z 2022-11-23T02:06:52.0428579Z OK 2022-11-23T02:06:52.0428584Z 2022-11-23T02:06:52.0428693Z Generating XML reports... 2022-11-23T02:06:52.0429133Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015842.xml 2022-11-23T02:06:52.0429448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0429829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0429994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0430381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0430559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0430564Z 2022-11-23T02:06:52.0430658Z Running tests... 2022-11-23T02:06:52.0430922Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0431216Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26687 2022-11-23T02:06:52.0431426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26688 2022-11-23T02:06:52.0431671Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0432055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0432219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0432607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0432783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0433008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0433380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0433542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0433925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0434159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0434385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0434788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0435185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0435400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0435615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0435701Z ok (5.020s) 2022-11-23T02:06:52.0435707Z 2022-11-23T02:06:52.0435974Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0436068Z Ran 1 test in 5.020s 2022-11-23T02:06:52.0436122Z 2022-11-23T02:06:52.0436199Z OK 2022-11-23T02:06:52.0436208Z 2022-11-23T02:06:52.0436317Z Generating XML reports... 2022-11-23T02:06:52.0436764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015852.xml 2022-11-23T02:06:52.0437080Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0437453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0437605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0437989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0438169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0438175Z 2022-11-23T02:06:52.0438270Z Running tests... 2022-11-23T02:06:52.0438536Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0438858Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26890 2022-11-23T02:06:52.0439065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26891 2022-11-23T02:06:52.0439322Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0439694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0439856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0440242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0440421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0440657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0441030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0441192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0441577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0441753Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0441977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0442378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0442774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0443042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0443266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:06:52.0443481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0443693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:06:52.0444094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0444487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0444572Z ok (5.025s) 2022-11-23T02:06:52.0444578Z 2022-11-23T02:06:52.0444840Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0444938Z Ran 1 test in 5.026s 2022-11-23T02:06:52.0444990Z 2022-11-23T02:06:52.0445068Z OK 2022-11-23T02:06:52.0445076Z 2022-11-23T02:06:52.0445187Z Generating XML reports... 2022-11-23T02:06:52.0445634Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015902.xml 2022-11-23T02:06:52.0445950Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0446324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0446486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0446870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0447049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0447054Z 2022-11-23T02:06:52.0447147Z Running tests... 2022-11-23T02:06:52.0447412Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0447725Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27095 2022-11-23T02:06:52.0447932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27096 2022-11-23T02:06:52.0448186Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0448558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0448720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0449102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0449269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0449495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0449871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0450032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0450421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0450649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0450878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0451279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0451674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0451953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0452177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:06:52.0452391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0452614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:06:52.0453024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0453417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:06:52.0453502Z ok (5.010s) 2022-11-23T02:06:52.0453509Z 2022-11-23T02:06:52.0453773Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0453869Z Ran 1 test in 5.010s 2022-11-23T02:06:52.0453938Z 2022-11-23T02:06:52.0454014Z OK 2022-11-23T02:06:52.0454024Z 2022-11-23T02:06:52.0454133Z Generating XML reports... 2022-11-23T02:06:52.0454584Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015911.xml 2022-11-23T02:06:52.0454899Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0455270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0455424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0455813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0455992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0455998Z 2022-11-23T02:06:52.0456091Z Running tests... 2022-11-23T02:06:52.0456355Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0456797Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.003s) 2022-11-23T02:06:52.0456804Z 2022-11-23T02:06:52.0457066Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0457162Z Ran 1 test in 0.003s 2022-11-23T02:06:52.0457168Z 2022-11-23T02:06:52.0457259Z OK (skipped=1) 2022-11-23T02:06:52.0457264Z 2022-11-23T02:06:52.0457372Z Generating XML reports... 2022-11-23T02:06:52.0457814Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015920.xml 2022-11-23T02:06:52.0458129Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0458502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0458664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0459056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0459233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0459239Z 2022-11-23T02:06:52.0459332Z Running tests... 2022-11-23T02:06:52.0459597Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0459820Z test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s) 2022-11-23T02:06:52.0459826Z 2022-11-23T02:06:52.0460087Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0460182Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0460188Z 2022-11-23T02:06:52.0460278Z OK (skipped=1) 2022-11-23T02:06:52.0460283Z 2022-11-23T02:06:52.0460392Z Generating XML reports... 2022-11-23T02:06:52.0460873Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015924.xml 2022-11-23T02:06:52.0461198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0461573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0461735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0462126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0462303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0462308Z 2022-11-23T02:06:52.0462404Z Running tests... 2022-11-23T02:06:52.0462671Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0462896Z test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:06:52.0462902Z 2022-11-23T02:06:52.0463301Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0463397Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0463403Z 2022-11-23T02:06:52.0463495Z OK (skipped=1) 2022-11-23T02:06:52.0463500Z 2022-11-23T02:06:52.0463611Z Generating XML reports... 2022-11-23T02:06:52.0464052Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015928.xml 2022-11-23T02:06:52.0464367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0464742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0464904Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0465287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0465463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0465473Z 2022-11-23T02:06:52.0465570Z Running tests... 2022-11-23T02:06:52.0465837Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0466087Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:06:52.0466094Z 2022-11-23T02:06:52.0466354Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0466441Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0466447Z 2022-11-23T02:06:52.0466541Z OK (skipped=1) 2022-11-23T02:06:52.0466546Z 2022-11-23T02:06:52.0466660Z Generating XML reports... 2022-11-23T02:06:52.0467105Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015932.xml 2022-11-23T02:06:52.0467420Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0467796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0467965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0468348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0468523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0468529Z 2022-11-23T02:06:52.0468623Z Running tests... 2022-11-23T02:06:52.0468888Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0469134Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s) 2022-11-23T02:06:52.0469140Z 2022-11-23T02:06:52.0469400Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0469495Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0469501Z 2022-11-23T02:06:52.0469592Z OK (skipped=1) 2022-11-23T02:06:52.0469600Z 2022-11-23T02:06:52.0469712Z Generating XML reports... 2022-11-23T02:06:52.0470207Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015936.xml 2022-11-23T02:06:52.0470528Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0470903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0471065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0471448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0471623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0471629Z 2022-11-23T02:06:52.0471713Z Running tests... 2022-11-23T02:06:52.0471982Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0472493Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s) 2022-11-23T02:06:52.0472500Z 2022-11-23T02:06:52.0472764Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0472859Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0472864Z 2022-11-23T02:06:52.0472956Z OK (skipped=1) 2022-11-23T02:06:52.0472962Z 2022-11-23T02:06:52.0473070Z Generating XML reports... 2022-11-23T02:06:52.0473511Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015940.xml 2022-11-23T02:06:52.0473828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0474205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0474370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0474764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0474939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0474945Z 2022-11-23T02:06:52.0475040Z Running tests... 2022-11-23T02:06:52.0475309Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0475803Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:06:52.0475809Z 2022-11-23T02:06:52.0476072Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0476171Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0476177Z 2022-11-23T02:06:52.0476268Z OK (skipped=1) 2022-11-23T02:06:52.0476274Z 2022-11-23T02:06:52.0476384Z Generating XML reports... 2022-11-23T02:06:52.0476828Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015944.xml 2022-11-23T02:06:52.0477146Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0477518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0477682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0478059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0478239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0478245Z 2022-11-23T02:06:52.0478342Z Running tests... 2022-11-23T02:06:52.0478606Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0479027Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:52.0479036Z 2022-11-23T02:06:52.0479348Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0479447Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0479452Z 2022-11-23T02:06:52.0479544Z OK (skipped=1) 2022-11-23T02:06:52.0479550Z 2022-11-23T02:06:52.0479659Z Generating XML reports... 2022-11-23T02:06:52.0480103Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015948.xml 2022-11-23T02:06:52.0480418Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0480790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0480953Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0481340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0481571Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0481577Z 2022-11-23T02:06:52.0481674Z Running tests... 2022-11-23T02:06:52.0481942Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0482341Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:52.0482347Z 2022-11-23T02:06:52.0482608Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0482706Z Ran 1 test in 0.003s 2022-11-23T02:06:52.0482712Z 2022-11-23T02:06:52.0482803Z OK (skipped=1) 2022-11-23T02:06:52.0482809Z 2022-11-23T02:06:52.0482918Z Generating XML reports... 2022-11-23T02:06:52.0483363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015952.xml 2022-11-23T02:06:52.0483668Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0484053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0484215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0484599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0484776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0484782Z 2022-11-23T02:06:52.0484877Z Running tests... 2022-11-23T02:06:52.0485144Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0485572Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:52.0485578Z 2022-11-23T02:06:52.0485839Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0485939Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0485944Z 2022-11-23T02:06:52.0486039Z OK (skipped=1) 2022-11-23T02:06:52.0486045Z 2022-11-23T02:06:52.0486154Z Generating XML reports... 2022-11-23T02:06:52.0486595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015956.xml 2022-11-23T02:06:52.0486910Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0487285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0487446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0487833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0488010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0488016Z 2022-11-23T02:06:52.0488108Z Running tests... 2022-11-23T02:06:52.0488426Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0488850Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:52.0488856Z 2022-11-23T02:06:52.0489120Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0489216Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0489222Z 2022-11-23T02:06:52.0489304Z OK (skipped=1) 2022-11-23T02:06:52.0489319Z 2022-11-23T02:06:52.0489419Z Generating XML reports... 2022-11-23T02:06:52.0489862Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020000.xml 2022-11-23T02:06:52.0490178Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0490672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0490914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0491309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0491482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0491488Z 2022-11-23T02:06:52.0491580Z Running tests... 2022-11-23T02:06:52.0491843Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0492262Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:52.0492268Z 2022-11-23T02:06:52.0492531Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0492626Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0492631Z 2022-11-23T02:06:52.0492723Z OK (skipped=1) 2022-11-23T02:06:52.0492729Z 2022-11-23T02:06:52.0492838Z Generating XML reports... 2022-11-23T02:06:52.0493286Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020004.xml 2022-11-23T02:06:52.0493600Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0493973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0494134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0494517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0494692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0494698Z 2022-11-23T02:06:52.0494790Z Running tests... 2022-11-23T02:06:52.0495055Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0495367Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28092 2022-11-23T02:06:52.0495580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28093 2022-11-23T02:06:52.0495838Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0496220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0496384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0496771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0496951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0497178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0497611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0497779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0498167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0498342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0498567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0498965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0499360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0499574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0499836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0500114Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0500387Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0500475Z ok (5.723s) 2022-11-23T02:06:52.0500481Z 2022-11-23T02:06:52.0500747Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0500845Z Ran 1 test in 5.724s 2022-11-23T02:06:52.0500851Z 2022-11-23T02:06:52.0500919Z OK 2022-11-23T02:06:52.0500936Z 2022-11-23T02:06:52.0501036Z Generating XML reports... 2022-11-23T02:06:52.0501481Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020008.xml 2022-11-23T02:06:52.0501796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0502172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0502347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0502730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0502904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0502910Z 2022-11-23T02:06:52.0503002Z Running tests... 2022-11-23T02:06:52.0503265Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0503583Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28306 2022-11-23T02:06:52.0503790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28307 2022-11-23T02:06:52.0504046Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0504427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0504589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0504969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0505144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0505367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0505738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0505901Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0506286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0506525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0506742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0507142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0507537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0507751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0507967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0508241Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0508517Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0508602Z ok (5.415s) 2022-11-23T02:06:52.0508656Z 2022-11-23T02:06:52.0508930Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0509030Z Ran 1 test in 5.415s 2022-11-23T02:06:52.0509036Z 2022-11-23T02:06:52.0509115Z OK 2022-11-23T02:06:52.0509121Z 2022-11-23T02:06:52.0509228Z Generating XML reports... 2022-11-23T02:06:52.0509670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020018.xml 2022-11-23T02:06:52.0509984Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0510361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0510522Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0510905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0511080Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0511091Z 2022-11-23T02:06:52.0511189Z Running tests... 2022-11-23T02:06:52.0511456Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0511776Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28520 2022-11-23T02:06:52.0511983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28521 2022-11-23T02:06:52.0512228Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0512605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0512770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0513160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0513347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0513574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0513947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0514110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0514495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0514672Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0514897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0515298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0515754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0515973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0516186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0516467Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0516745Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0516830Z ok (5.413s) 2022-11-23T02:06:52.0516836Z 2022-11-23T02:06:52.0517099Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0517198Z Ran 1 test in 5.414s 2022-11-23T02:06:52.0517203Z 2022-11-23T02:06:52.0517279Z OK 2022-11-23T02:06:52.0517284Z 2022-11-23T02:06:52.0517392Z Generating XML reports... 2022-11-23T02:06:52.0517826Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020027.xml 2022-11-23T02:06:52.0518195Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0518570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0518733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0519117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0519299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0519305Z 2022-11-23T02:06:52.0519397Z Running tests... 2022-11-23T02:06:52.0519661Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0519975Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28734 2022-11-23T02:06:52.0520191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28735 2022-11-23T02:06:52.0520452Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0520829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0520993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0521381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0521556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0521782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0522158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0522327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0522715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0522894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0523120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0523525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0523919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0524126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0524345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0524688Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0524970Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0525056Z ok (6.313s) 2022-11-23T02:06:52.0525062Z 2022-11-23T02:06:52.0525326Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0525423Z Ran 1 test in 6.313s 2022-11-23T02:06:52.0525429Z 2022-11-23T02:06:52.0525511Z OK 2022-11-23T02:06:52.0525517Z 2022-11-23T02:06:52.0525626Z Generating XML reports... 2022-11-23T02:06:52.0526068Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020037.xml 2022-11-23T02:06:52.0526382Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0526757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0526978Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0527368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0527544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0527551Z 2022-11-23T02:06:52.0527645Z Running tests... 2022-11-23T02:06:52.0527913Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0528845Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.583s) 2022-11-23T02:06:52.0528852Z 2022-11-23T02:06:52.0529116Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0529220Z Ran 1 test in 0.583s 2022-11-23T02:06:52.0529227Z 2022-11-23T02:06:52.0529322Z OK (skipped=1) 2022-11-23T02:06:52.0529327Z 2022-11-23T02:06:52.0529440Z Generating XML reports... 2022-11-23T02:06:52.0529884Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020047.xml 2022-11-23T02:06:52.0530200Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0530645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0530809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0531196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0531377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0531388Z 2022-11-23T02:06:52.0531481Z Running tests... 2022-11-23T02:06:52.0531751Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0531987Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:06:52.0531994Z 2022-11-23T02:06:52.0532257Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0532354Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0532360Z 2022-11-23T02:06:52.0532452Z OK (skipped=1) 2022-11-23T02:06:52.0532457Z 2022-11-23T02:06:52.0532565Z Generating XML reports... 2022-11-23T02:06:52.0533007Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020052.xml 2022-11-23T02:06:52.0533321Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0533693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0533923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0534312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0534488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0534494Z 2022-11-23T02:06:52.0534587Z Running tests... 2022-11-23T02:06:52.0534851Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0535114Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:06:52.0535119Z 2022-11-23T02:06:52.0535381Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0535480Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0535485Z 2022-11-23T02:06:52.0535568Z OK (skipped=1) 2022-11-23T02:06:52.0535586Z 2022-11-23T02:06:52.0535686Z Generating XML reports... 2022-11-23T02:06:52.0536190Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020056.xml 2022-11-23T02:06:52.0536504Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0536878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0537039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0537428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0537604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0537609Z 2022-11-23T02:06:52.0537707Z Running tests... 2022-11-23T02:06:52.0537971Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0538272Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:06:52.0538284Z 2022-11-23T02:06:52.0538546Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0538642Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0538648Z 2022-11-23T02:06:52.0538742Z OK (skipped=1) 2022-11-23T02:06:52.0538747Z 2022-11-23T02:06:52.0538860Z Generating XML reports... 2022-11-23T02:06:52.0539298Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020100.xml 2022-11-23T02:06:52.0539615Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0539990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0540153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0540542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0540722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0540727Z 2022-11-23T02:06:52.0540822Z Running tests... 2022-11-23T02:06:52.0541088Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0541425Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29210 2022-11-23T02:06:52.0541636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29211 2022-11-23T02:06:52.0541893Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0542267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0542427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0542869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0543051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0543275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0543656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0543820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0544205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0544382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0544609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0545062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0545458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0545673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0545890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0545975Z ok (5.108s) 2022-11-23T02:06:52.0545981Z 2022-11-23T02:06:52.0546245Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0546343Z Ran 1 test in 5.109s 2022-11-23T02:06:52.0546349Z 2022-11-23T02:06:52.0546427Z OK 2022-11-23T02:06:52.0546433Z 2022-11-23T02:06:52.0546541Z Generating XML reports... 2022-11-23T02:06:52.0546986Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020104.xml 2022-11-23T02:06:52.0547298Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0547671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0547833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0548221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0548397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0548403Z 2022-11-23T02:06:52.0548495Z Running tests... 2022-11-23T02:06:52.0548758Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0549094Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29413 2022-11-23T02:06:52.0549307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29414 2022-11-23T02:06:52.0549565Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0549941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0550101Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0550483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0550662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0550893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0551265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0551480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0551872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0552050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0552273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0552672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0553068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0553287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0553494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0553653Z ok (5.012s) 2022-11-23T02:06:52.0553663Z 2022-11-23T02:06:52.0553933Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0554030Z Ran 1 test in 5.012s 2022-11-23T02:06:52.0554036Z 2022-11-23T02:06:52.0554120Z OK 2022-11-23T02:06:52.0554125Z 2022-11-23T02:06:52.0554238Z Generating XML reports... 2022-11-23T02:06:52.0554682Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020113.xml 2022-11-23T02:06:52.0554995Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0555371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0555534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0555921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0556108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0556114Z 2022-11-23T02:06:52.0556207Z Running tests... 2022-11-23T02:06:52.0556472Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0556741Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:06:52.0556746Z 2022-11-23T02:06:52.0557011Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0557111Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0557117Z 2022-11-23T02:06:52.0557212Z OK (skipped=1) 2022-11-23T02:06:52.0557218Z 2022-11-23T02:06:52.0557330Z Generating XML reports... 2022-11-23T02:06:52.0557770Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020122.xml 2022-11-23T02:06:52.0558086Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0558470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0558623Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0559008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0559185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0559191Z 2022-11-23T02:06:52.0559283Z Running tests... 2022-11-23T02:06:52.0559546Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0559833Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:06:52.0559839Z 2022-11-23T02:06:52.0560104Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0560209Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0560214Z 2022-11-23T02:06:52.0560357Z OK (skipped=1) 2022-11-23T02:06:52.0560364Z 2022-11-23T02:06:52.0560475Z Generating XML reports... 2022-11-23T02:06:52.0560922Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020126.xml 2022-11-23T02:06:52.0561239Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0561613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0561776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0562162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0562340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0562346Z 2022-11-23T02:06:52.0562439Z Running tests... 2022-11-23T02:06:52.0562759Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0563703Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.635s) 2022-11-23T02:06:52.0563710Z 2022-11-23T02:06:52.0563972Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0564069Z Ran 1 test in 0.636s 2022-11-23T02:06:52.0564074Z 2022-11-23T02:06:52.0564166Z OK (skipped=1) 2022-11-23T02:06:52.0564172Z 2022-11-23T02:06:52.0564281Z Generating XML reports... 2022-11-23T02:06:52.0564723Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020131.xml 2022-11-23T02:06:52.0565047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0565423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0565575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0565963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0566143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0566148Z 2022-11-23T02:06:52.0566241Z Running tests... 2022-11-23T02:06:52.0566506Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0566831Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29814 2022-11-23T02:06:52.0567041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29815 2022-11-23T02:06:52.0567307Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0567680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0567841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0568227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0568406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0568629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0569000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0569161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0569599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0569776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0570000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0570405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0570921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0571139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0571353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0571591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hw4rjff 2022-11-23T02:06:52.0571898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hw4rjff/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0572132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5iogaqju 2022-11-23T02:06:52.0572383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5iogaqju/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0572671Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0572945Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0573035Z ok (7.523s) 2022-11-23T02:06:52.0573041Z 2022-11-23T02:06:52.0573307Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0573405Z Ran 1 test in 7.523s 2022-11-23T02:06:52.0573410Z 2022-11-23T02:06:52.0573488Z OK 2022-11-23T02:06:52.0573493Z 2022-11-23T02:06:52.0573601Z Generating XML reports... 2022-11-23T02:06:52.0574051Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020135.xml 2022-11-23T02:06:52.0574367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0574743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0574908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0575295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0575475Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0575480Z 2022-11-23T02:06:52.0575574Z Running tests... 2022-11-23T02:06:52.0575840Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0576161Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30034 2022-11-23T02:06:52.0576372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30035 2022-11-23T02:06:52.0576630Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0577003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0577155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0577538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0577716Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0577941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0578369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0578536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0578923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0579106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0579329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0579729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0580125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0580342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0580561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0580887Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0581162Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0581250Z ok (5.717s) 2022-11-23T02:06:52.0581256Z 2022-11-23T02:06:52.0581518Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0581614Z Ran 1 test in 5.718s 2022-11-23T02:06:52.0581619Z 2022-11-23T02:06:52.0581695Z OK 2022-11-23T02:06:52.0581701Z 2022-11-23T02:06:52.0581809Z Generating XML reports... 2022-11-23T02:06:52.0582252Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020147.xml 2022-11-23T02:06:52.0582557Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0582933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0583103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0583490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0583665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0583671Z 2022-11-23T02:06:52.0583765Z Running tests... 2022-11-23T02:06:52.0584031Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0584363Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30248 2022-11-23T02:06:52.0584569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30249 2022-11-23T02:06:52.0584824Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0585203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0585366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0585754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0585929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0586155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0586526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0586688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0587071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0587250Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0587528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0587927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0588325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0588541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0588746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0589023Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0589296Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0589384Z ok (5.715s) 2022-11-23T02:06:52.0589389Z 2022-11-23T02:06:52.0589720Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0589819Z Ran 1 test in 5.715s 2022-11-23T02:06:52.0589824Z 2022-11-23T02:06:52.0589901Z OK 2022-11-23T02:06:52.0589906Z 2022-11-23T02:06:52.0590013Z Generating XML reports... 2022-11-23T02:06:52.0590457Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020157.xml 2022-11-23T02:06:52.0590774Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0591149Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0591310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0591695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0591877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0591887Z 2022-11-23T02:06:52.0591987Z Running tests... 2022-11-23T02:06:52.0592255Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0593203Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.579s) 2022-11-23T02:06:52.0593210Z 2022-11-23T02:06:52.0593478Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0593577Z Ran 1 test in 0.579s 2022-11-23T02:06:52.0593582Z 2022-11-23T02:06:52.0593675Z OK (skipped=1) 2022-11-23T02:06:52.0593681Z 2022-11-23T02:06:52.0593789Z Generating XML reports... 2022-11-23T02:06:52.0594241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020207.xml 2022-11-23T02:06:52.0594559Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0594933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0595098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0595474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0595652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0595657Z 2022-11-23T02:06:52.0595749Z Running tests... 2022-11-23T02:06:52.0596014Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0597032Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.578s) 2022-11-23T02:06:52.0597044Z 2022-11-23T02:06:52.0597316Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0597418Z Ran 1 test in 0.579s 2022-11-23T02:06:52.0597423Z 2022-11-23T02:06:52.0597515Z OK (skipped=1) 2022-11-23T02:06:52.0597521Z 2022-11-23T02:06:52.0597633Z Generating XML reports... 2022-11-23T02:06:52.0598075Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020211.xml 2022-11-23T02:06:52.0598391Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0598765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0598983Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0599376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0599553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0599559Z 2022-11-23T02:06:52.0599650Z Running tests... 2022-11-23T02:06:52.0599914Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0600901Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75052 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.577s) 2022-11-23T02:06:52.0600909Z 2022-11-23T02:06:52.0601177Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0601275Z Ran 1 test in 0.578s 2022-11-23T02:06:52.0601281Z 2022-11-23T02:06:52.0601372Z OK (skipped=1) 2022-11-23T02:06:52.0601377Z 2022-11-23T02:06:52.0601489Z Generating XML reports... 2022-11-23T02:06:52.0601929Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020216.xml 2022-11-23T02:06:52.0602246Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0602621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0602784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0603169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0603349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0603358Z 2022-11-23T02:06:52.0603456Z Running tests... 2022-11-23T02:06:52.0603712Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0604725Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75139 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.640s) 2022-11-23T02:06:52.0604744Z 2022-11-23T02:06:52.0605008Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0605095Z Ran 1 test in 0.640s 2022-11-23T02:06:52.0605113Z 2022-11-23T02:06:52.0605195Z OK (skipped=1) 2022-11-23T02:06:52.0605200Z 2022-11-23T02:06:52.0605312Z Generating XML reports... 2022-11-23T02:06:52.0605815Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020221.xml 2022-11-23T02:06:52.0606147Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0606528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0606691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0607078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0607255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0607261Z 2022-11-23T02:06:52.0607355Z Running tests... 2022-11-23T02:06:52.0607621Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0608575Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.581s) 2022-11-23T02:06:52.0608626Z 2022-11-23T02:06:52.0608898Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0608997Z Ran 1 test in 0.581s 2022-11-23T02:06:52.0609003Z 2022-11-23T02:06:52.0609099Z OK (skipped=1) 2022-11-23T02:06:52.0609105Z 2022-11-23T02:06:52.0609213Z Generating XML reports... 2022-11-23T02:06:52.0609659Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020225.xml 2022-11-23T02:06:52.0609978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0610351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0610572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0610965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0611143Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0611149Z 2022-11-23T02:06:52.0611246Z Running tests... 2022-11-23T02:06:52.0611510Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0611771Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0611777Z 2022-11-23T02:06:52.0612040Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0612126Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0612144Z 2022-11-23T02:06:52.0612226Z OK (skipped=1) 2022-11-23T02:06:52.0612231Z 2022-11-23T02:06:52.0612342Z Generating XML reports... 2022-11-23T02:06:52.0612792Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020230.xml 2022-11-23T02:06:52.0613106Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0613483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0613647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0614031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0614209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0614215Z 2022-11-23T02:06:52.0614307Z Running tests... 2022-11-23T02:06:52.0614573Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0614889Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0614899Z 2022-11-23T02:06:52.0615169Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0615267Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0615273Z 2022-11-23T02:06:52.0615364Z OK (skipped=1) 2022-11-23T02:06:52.0615370Z 2022-11-23T02:06:52.0615478Z Generating XML reports... 2022-11-23T02:06:52.0615920Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020234.xml 2022-11-23T02:06:52.0616233Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0616606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0616766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0617148Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0617388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0617393Z 2022-11-23T02:06:52.0617490Z Running tests... 2022-11-23T02:06:52.0617749Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0618013Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0618018Z 2022-11-23T02:06:52.0618284Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0618382Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0618387Z 2022-11-23T02:06:52.0618485Z OK (skipped=1) 2022-11-23T02:06:52.0618490Z 2022-11-23T02:06:52.0618602Z Generating XML reports... 2022-11-23T02:06:52.0619046Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020238.xml 2022-11-23T02:06:52.0619365Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0619746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0619910Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0620294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0620471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0620477Z 2022-11-23T02:06:52.0620578Z Running tests... 2022-11-23T02:06:52.0620845Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0621104Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0621110Z 2022-11-23T02:06:52.0621371Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0621472Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0621477Z 2022-11-23T02:06:52.0621573Z OK (skipped=1) 2022-11-23T02:06:52.0621578Z 2022-11-23T02:06:52.0621686Z Generating XML reports... 2022-11-23T02:06:52.0622128Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020242.xml 2022-11-23T02:06:52.0622444Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0622818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0622972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0623357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0623537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0623543Z 2022-11-23T02:06:52.0623634Z Running tests... 2022-11-23T02:06:52.0623953Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0624205Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0624211Z 2022-11-23T02:06:52.0624477Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0624577Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0624583Z 2022-11-23T02:06:52.0624674Z OK (skipped=1) 2022-11-23T02:06:52.0624679Z 2022-11-23T02:06:52.0624787Z Generating XML reports... 2022-11-23T02:06:52.0625230Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020246.xml 2022-11-23T02:06:52.0625544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0625918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0626133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0626522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0626700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0626706Z 2022-11-23T02:06:52.0626801Z Running tests... 2022-11-23T02:06:52.0627065Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0627312Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0627318Z 2022-11-23T02:06:52.0627584Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0627680Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0627685Z 2022-11-23T02:06:52.0627777Z OK (skipped=1) 2022-11-23T02:06:52.0627783Z 2022-11-23T02:06:52.0627894Z Generating XML reports... 2022-11-23T02:06:52.0628329Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020250.xml 2022-11-23T02:06:52.0628649Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0629024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0629188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0629573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0629754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0629760Z 2022-11-23T02:06:52.0629855Z Running tests... 2022-11-23T02:06:52.0630121Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0630383Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0630392Z 2022-11-23T02:06:52.0630656Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0630754Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0630759Z 2022-11-23T02:06:52.0630852Z OK (skipped=1) 2022-11-23T02:06:52.0630858Z 2022-11-23T02:06:52.0630971Z Generating XML reports... 2022-11-23T02:06:52.0631412Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020254.xml 2022-11-23T02:06:52.0631729Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0632105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0632269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0632656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0632888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0632895Z 2022-11-23T02:06:52.0632992Z Running tests... 2022-11-23T02:06:52.0633265Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0633518Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0633524Z 2022-11-23T02:06:52.0633790Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0633878Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0633883Z 2022-11-23T02:06:52.0633979Z OK (skipped=1) 2022-11-23T02:06:52.0633985Z 2022-11-23T02:06:52.0634097Z Generating XML reports... 2022-11-23T02:06:52.0634538Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020258.xml 2022-11-23T02:06:52.0634855Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0635286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0635454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0635840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0636017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0636023Z 2022-11-23T02:06:52.0636119Z Running tests... 2022-11-23T02:06:52.0636389Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0636629Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0636635Z 2022-11-23T02:06:52.0636899Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0636996Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0637001Z 2022-11-23T02:06:52.0637098Z OK (skipped=1) 2022-11-23T02:06:52.0637103Z 2022-11-23T02:06:52.0637217Z Generating XML reports... 2022-11-23T02:06:52.0637656Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020302.xml 2022-11-23T02:06:52.0637972Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0638347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0638507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0638889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0639066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0639072Z 2022-11-23T02:06:52.0639156Z Running tests... 2022-11-23T02:06:52.0639418Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0639667Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0639673Z 2022-11-23T02:06:52.0639937Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0640035Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0640041Z 2022-11-23T02:06:52.0640135Z OK (skipped=1) 2022-11-23T02:06:52.0640141Z 2022-11-23T02:06:52.0640252Z Generating XML reports... 2022-11-23T02:06:52.0640691Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020306.xml 2022-11-23T02:06:52.0641006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0641381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0641541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0641983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0642162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0642168Z 2022-11-23T02:06:52.0642260Z Running tests... 2022-11-23T02:06:52.0642533Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0642841Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31452 2022-11-23T02:06:52.0643048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31453 2022-11-23T02:06:52.0643305Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0643681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0643891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0644281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0644458Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0644677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0645054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0645215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0645601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0645779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0646004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0646413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0646810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0647027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0647245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0647588Z STAGE:2022-11-23 02:03:13 31452:31452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0647867Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0648206Z STAGE:2022-11-23 02:03:13 31453:31453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0648482Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0648827Z STAGE:2022-11-23 02:03:13 31453:31453 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0649181Z STAGE:2022-11-23 02:03:13 31453:31453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0649406Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0649747Z STAGE:2022-11-23 02:03:13 31452:31452 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0650100Z STAGE:2022-11-23 02:03:13 31452:31452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0650324Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0650708Z STAGE:2022-11-23 02:03:13 31452:31452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0651038Z STAGE:2022-11-23 02:03:13 31453:31453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0651443Z STAGE:2022-11-23 02:03:13 31453:31453 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0651791Z STAGE:2022-11-23 02:03:13 31453:31453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0652129Z STAGE:2022-11-23 02:03:13 31452:31452 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0652483Z STAGE:2022-11-23 02:03:13 31452:31452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0653274Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:06:52.0653369Z warnings.warn( 2022-11-23T02:06:52.0654149Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:06:52.0654303Z warnings.warn( 2022-11-23T02:06:52.0654391Z ok (5.408s) 2022-11-23T02:06:52.0654397Z 2022-11-23T02:06:52.0654667Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0654762Z Ran 1 test in 5.408s 2022-11-23T02:06:52.0654768Z 2022-11-23T02:06:52.0654846Z OK 2022-11-23T02:06:52.0654851Z 2022-11-23T02:06:52.0654963Z Generating XML reports... 2022-11-23T02:06:52.0655409Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020310.xml 2022-11-23T02:06:52.0655728Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0656107Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0656280Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0656675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0656854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0656860Z 2022-11-23T02:06:52.0656956Z Running tests... 2022-11-23T02:06:52.0657226Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0657475Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0657481Z 2022-11-23T02:06:52.0657745Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0657840Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0657846Z 2022-11-23T02:06:52.0657937Z OK (skipped=1) 2022-11-23T02:06:52.0657943Z 2022-11-23T02:06:52.0658055Z Generating XML reports... 2022-11-23T02:06:52.0658490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020320.xml 2022-11-23T02:06:52.0658806Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0659182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0659343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0659732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0659914Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0659920Z 2022-11-23T02:06:52.0660017Z Running tests... 2022-11-23T02:06:52.0660286Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0660673Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31734 2022-11-23T02:06:52.0660891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31735 2022-11-23T02:06:52.0661150Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0661532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0661694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0662080Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0662255Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0662483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0662914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0663080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0663471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0663651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0663878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0664284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0664688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0664896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0665122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0665461Z STAGE:2022-11-23 02:03:27 31734:31734 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0665742Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0666076Z STAGE:2022-11-23 02:03:27 31735:31735 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0666352Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0666693Z STAGE:2022-11-23 02:03:27 31734:31734 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0667030Z STAGE:2022-11-23 02:03:27 31735:31735 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0667386Z STAGE:2022-11-23 02:03:27 31734:31734 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0667739Z STAGE:2022-11-23 02:03:27 31735:31735 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0667968Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0668191Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0668523Z STAGE:2022-11-23 02:03:27 31735:31735 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0668862Z STAGE:2022-11-23 02:03:27 31735:31735 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0669215Z STAGE:2022-11-23 02:03:27 31735:31735 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0669545Z STAGE:2022-11-23 02:03:27 31734:31734 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0669886Z STAGE:2022-11-23 02:03:27 31734:31734 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0670302Z STAGE:2022-11-23 02:03:27 31734:31734 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0670396Z ok (5.512s) 2022-11-23T02:06:52.0670402Z 2022-11-23T02:06:52.0670676Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0670772Z Ran 1 test in 5.513s 2022-11-23T02:06:52.0670778Z 2022-11-23T02:06:52.0670861Z OK 2022-11-23T02:06:52.0670867Z 2022-11-23T02:06:52.0670965Z Generating XML reports... 2022-11-23T02:06:52.0671412Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020324.xml 2022-11-23T02:06:52.0671729Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0672110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0672270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0672665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0672890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0672896Z 2022-11-23T02:06:52.0672992Z Running tests... 2022-11-23T02:06:52.0673264Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0673584Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31950 2022-11-23T02:06:52.0673796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31951 2022-11-23T02:06:52.0674052Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0674431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0674593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0674991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0675166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0675391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0675766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0675927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0676314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0676488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0676714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0677124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0677514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0677729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0677945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0678219Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0678493Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0678581Z ok (5.450s) 2022-11-23T02:06:52.0678587Z 2022-11-23T02:06:52.0678853Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0678950Z Ran 1 test in 5.451s 2022-11-23T02:06:52.0678956Z 2022-11-23T02:06:52.0679035Z OK 2022-11-23T02:06:52.0679041Z 2022-11-23T02:06:52.0679199Z Generating XML reports... 2022-11-23T02:06:52.0679656Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020334.xml 2022-11-23T02:06:52.0679973Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0680347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0680507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0680897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0681074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0681080Z 2022-11-23T02:06:52.0681175Z Running tests... 2022-11-23T02:06:52.0681442Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0681735Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0681741Z 2022-11-23T02:06:52.0682007Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0682102Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0682108Z 2022-11-23T02:06:52.0682190Z OK (skipped=1) 2022-11-23T02:06:52.0682211Z 2022-11-23T02:06:52.0682311Z Generating XML reports... 2022-11-23T02:06:52.0682758Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020343.xml 2022-11-23T02:06:52.0683076Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0683450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0683610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0684005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0684181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0684187Z 2022-11-23T02:06:52.0684278Z Running tests... 2022-11-23T02:06:52.0684545Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0684850Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32230 2022-11-23T02:06:52.0685056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32231 2022-11-23T02:06:52.0685312Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0685686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0685856Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0686248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0686422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0686651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0687027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0687194Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0687585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0687765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0687993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0688437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0688841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0689060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0689278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0689620Z STAGE:2022-11-23 02:03:50 32231:32231 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0689953Z STAGE:2022-11-23 02:03:50 32230:32230 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0690233Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0690509Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0691060Z STAGE:2022-11-23 02:03:50 32231:32231 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0691637Z STAGE:2022-11-23 02:03:50 32230:32230 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:03:50 32231:32231 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0691643Z 2022-11-23T02:06:52.0691865Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0692222Z STAGE:2022-11-23 02:03:50 32230:32230 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0692444Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0692779Z STAGE:2022-11-23 02:03:50 32231:32231 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0693115Z STAGE:2022-11-23 02:03:50 32231:32231 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0693478Z STAGE:2022-11-23 02:03:50 32231:32231 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0693809Z STAGE:2022-11-23 02:03:50 32230:32230 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0694145Z STAGE:2022-11-23 02:03:50 32230:32230 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0694494Z STAGE:2022-11-23 02:03:50 32230:32230 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0694580Z ok (5.679s) 2022-11-23T02:06:52.0694586Z 2022-11-23T02:06:52.0694851Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0694946Z Ran 1 test in 5.679s 2022-11-23T02:06:52.0694951Z 2022-11-23T02:06:52.0695026Z OK 2022-11-23T02:06:52.0695032Z 2022-11-23T02:06:52.0695140Z Generating XML reports... 2022-11-23T02:06:52.0695578Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020347.xml 2022-11-23T02:06:52.0695901Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0696277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0696438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0696827Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0697002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0697008Z 2022-11-23T02:06:52.0697099Z Running tests... 2022-11-23T02:06:52.0697365Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0697673Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32446 2022-11-23T02:06:52.0697942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32447 2022-11-23T02:06:52.0698203Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0698587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0698752Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0699141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0699322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0699552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0699929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0700144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0700534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0700711Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0700938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0701342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0701746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0701954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0702171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0702517Z STAGE:2022-11-23 02:04:00 32446:32446 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0702793Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0703125Z STAGE:2022-11-23 02:04:00 32447:32447 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0703399Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0703740Z STAGE:2022-11-23 02:04:00 32446:32446 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0704094Z STAGE:2022-11-23 02:04:00 32446:32446 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0704436Z STAGE:2022-11-23 02:04:00 32447:32447 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0704663Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0705020Z STAGE:2022-11-23 02:04:00 32447:32447 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0705246Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0705584Z STAGE:2022-11-23 02:04:00 32446:32446 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0705920Z STAGE:2022-11-23 02:04:00 32446:32446 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0706269Z STAGE:2022-11-23 02:04:00 32446:32446 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0706597Z STAGE:2022-11-23 02:04:00 32447:32447 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0706931Z STAGE:2022-11-23 02:04:00 32447:32447 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0707284Z STAGE:2022-11-23 02:04:00 32447:32447 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0707376Z ok (5.524s) 2022-11-23T02:06:52.0707381Z 2022-11-23T02:06:52.0707696Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0707795Z Ran 1 test in 5.524s 2022-11-23T02:06:52.0707801Z 2022-11-23T02:06:52.0707878Z OK 2022-11-23T02:06:52.0707884Z 2022-11-23T02:06:52.0707983Z Generating XML reports... 2022-11-23T02:06:52.0708432Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020357.xml 2022-11-23T02:06:52.0708747Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0709125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0709289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0709675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0709903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0709909Z 2022-11-23T02:06:52.0710003Z Running tests... 2022-11-23T02:06:52.0710275Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0710524Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0710530Z 2022-11-23T02:06:52.0710795Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0710893Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0710898Z 2022-11-23T02:06:52.0710990Z OK (skipped=1) 2022-11-23T02:06:52.0710996Z 2022-11-23T02:06:52.0711104Z Generating XML reports... 2022-11-23T02:06:52.0711549Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020407.xml 2022-11-23T02:06:52.0711867Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0712253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0712416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0712805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0712986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0712992Z 2022-11-23T02:06:52.0713086Z Running tests... 2022-11-23T02:06:52.0713356Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0713583Z test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:52.0713599Z 2022-11-23T02:06:52.0713854Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0713950Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0713956Z 2022-11-23T02:06:52.0714055Z OK (skipped=1) 2022-11-23T02:06:52.0714061Z 2022-11-23T02:06:52.0714176Z Generating XML reports... 2022-11-23T02:06:52.0714627Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020411.xml 2022-11-23T02:06:52.0714942Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0715321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0715486Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0715875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0716059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0716065Z 2022-11-23T02:06:52.0716161Z Running tests... 2022-11-23T02:06:52.0716430Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0716742Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0716749Z 2022-11-23T02:06:52.0717020Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0717119Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0717124Z 2022-11-23T02:06:52.0717219Z OK (skipped=1) 2022-11-23T02:06:52.0717224Z 2022-11-23T02:06:52.0717335Z Generating XML reports... 2022-11-23T02:06:52.0717785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020415.xml 2022-11-23T02:06:52.0718104Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0718482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0718647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0719090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0719256Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0719275Z 2022-11-23T02:06:52.0719358Z Running tests... 2022-11-23T02:06:52.0719629Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0719882Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:06:52.0719887Z 2022-11-23T02:06:52.0720154Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0720254Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0720259Z 2022-11-23T02:06:52.0720358Z OK (skipped=1) 2022-11-23T02:06:52.0720364Z 2022-11-23T02:06:52.0720475Z Generating XML reports... 2022-11-23T02:06:52.0720924Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020419.xml 2022-11-23T02:06:52.0721252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0721633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0721796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0722186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0722367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0722373Z 2022-11-23T02:06:52.0722472Z Running tests... 2022-11-23T02:06:52.0722740Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0723040Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32926 2022-11-23T02:06:52.0723250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32927 2022-11-23T02:06:52.0723517Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0723895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0724061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0724453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0724634Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0724848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0725225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0725389Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0725830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0726012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0726243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0726651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0727051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0727270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0727484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0727826Z STAGE:2022-11-23 02:04:26 32926:32926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0728166Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0728496Z STAGE:2022-11-23 02:04:26 32927:32927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0728774Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0729116Z STAGE:2022-11-23 02:04:26 32926:32926 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0729473Z STAGE:2022-11-23 02:04:26 32926:32926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0729700Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0730041Z STAGE:2022-11-23 02:04:26 32927:32927 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0730393Z STAGE:2022-11-23 02:04:26 32927:32927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0730676Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0731017Z STAGE:2022-11-23 02:04:26 32926:32926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0731352Z STAGE:2022-11-23 02:04:26 32926:32926 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0731702Z STAGE:2022-11-23 02:04:26 32926:32926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0732022Z STAGE:2022-11-23 02:04:26 32927:32927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0732359Z STAGE:2022-11-23 02:04:26 32927:32927 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0732709Z STAGE:2022-11-23 02:04:26 32927:32927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0732799Z ok (5.316s) 2022-11-23T02:06:52.0732808Z 2022-11-23T02:06:52.0733079Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0733174Z Ran 1 test in 5.316s 2022-11-23T02:06:52.0733180Z 2022-11-23T02:06:52.0733260Z OK 2022-11-23T02:06:52.0733265Z 2022-11-23T02:06:52.0733373Z Generating XML reports... 2022-11-23T02:06:52.0733818Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020423.xml 2022-11-23T02:06:52.0734134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0734514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0734679Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0735065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0735241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0735336Z 2022-11-23T02:06:52.0735433Z Running tests... 2022-11-23T02:06:52.0735705Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0736015Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33142 2022-11-23T02:06:52.0736226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33143 2022-11-23T02:06:52.0736483Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0736858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0737025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0737414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0737642Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0737873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0738252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0738412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0738796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0738972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0739197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0739595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0739997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0740211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0740431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0740766Z STAGE:2022-11-23 02:04:35 33142:33142 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0741095Z STAGE:2022-11-23 02:04:35 33143:33143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0741369Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0741641Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0741978Z STAGE:2022-11-23 02:04:35 33142:33142 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0742332Z STAGE:2022-11-23 02:04:35 33142:33142 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0742563Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0742902Z STAGE:2022-11-23 02:04:35 33143:33143 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0743252Z STAGE:2022-11-23 02:04:35 33143:33143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0743472Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0743801Z STAGE:2022-11-23 02:04:35 33142:33142 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0744132Z STAGE:2022-11-23 02:04:35 33143:33143 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0744460Z STAGE:2022-11-23 02:04:35 33142:33142 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0744862Z STAGE:2022-11-23 02:04:35 33142:33142 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0745202Z STAGE:2022-11-23 02:04:35 33143:33143 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0745554Z STAGE:2022-11-23 02:04:35 33143:33143 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0745640Z ok (5.245s) 2022-11-23T02:06:52.0745646Z 2022-11-23T02:06:52.0745910Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0746005Z Ran 1 test in 5.245s 2022-11-23T02:06:52.0746011Z 2022-11-23T02:06:52.0746090Z OK 2022-11-23T02:06:52.0746095Z 2022-11-23T02:06:52.0746204Z Generating XML reports... 2022-11-23T02:06:52.0746645Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020432.xml 2022-11-23T02:06:52.0746958Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0747390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0747550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0747934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0748110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0748116Z 2022-11-23T02:06:52.0748207Z Running tests... 2022-11-23T02:06:52.0748470Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0748724Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:52.0748730Z 2022-11-23T02:06:52.0748992Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0749090Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0749096Z 2022-11-23T02:06:52.0749191Z OK (skipped=1) 2022-11-23T02:06:52.0749196Z 2022-11-23T02:06:52.0749313Z Generating XML reports... 2022-11-23T02:06:52.0749748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020441.xml 2022-11-23T02:06:52.0750063Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0750439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0750599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0750986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0751163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0751169Z 2022-11-23T02:06:52.0751267Z Running tests... 2022-11-23T02:06:52.0751529Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0751781Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:06:52.0751787Z 2022-11-23T02:06:52.0752049Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0752147Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0752152Z 2022-11-23T02:06:52.0752244Z OK (skipped=1) 2022-11-23T02:06:52.0752250Z 2022-11-23T02:06:52.0752360Z Generating XML reports... 2022-11-23T02:06:52.0752798Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020445.xml 2022-11-23T02:06:52.0753110Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0753484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0753644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0754157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0754337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0754343Z 2022-11-23T02:06:52.0754436Z Running tests... 2022-11-23T02:06:52.0754701Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0755090Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:06:52.0755096Z 2022-11-23T02:06:52.0755356Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0755444Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0755461Z 2022-11-23T02:06:52.0755544Z OK (skipped=1) 2022-11-23T02:06:52.0755549Z 2022-11-23T02:06:52.0755657Z Generating XML reports... 2022-11-23T02:06:52.0756099Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020449.xml 2022-11-23T02:06:52.0756469Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0756840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0757004Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0757391Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0757570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0757576Z 2022-11-23T02:06:52.0757668Z Running tests... 2022-11-23T02:06:52.0757932Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0758184Z test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:06:52.0758190Z 2022-11-23T02:06:52.0758467Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0758563Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0758568Z 2022-11-23T02:06:52.0758659Z OK (skipped=1) 2022-11-23T02:06:52.0758665Z 2022-11-23T02:06:52.0758773Z Generating XML reports... 2022-11-23T02:06:52.0759213Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020454.xml 2022-11-23T02:06:52.0759525Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0759899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0760061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0760444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0760619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0760628Z 2022-11-23T02:06:52.0760722Z Running tests... 2022-11-23T02:06:52.0760976Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0761247Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.001s) 2022-11-23T02:06:52.0761253Z 2022-11-23T02:06:52.0761515Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0761613Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0761618Z 2022-11-23T02:06:52.0761713Z OK (skipped=1) 2022-11-23T02:06:52.0761719Z 2022-11-23T02:06:52.0761831Z Generating XML reports... 2022-11-23T02:06:52.0762280Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020458.xml 2022-11-23T02:06:52.0762597Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0763025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0763193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0763585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0763763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0763769Z 2022-11-23T02:06:52.0763862Z Running tests... 2022-11-23T02:06:52.0764126Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0764424Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.001s) 2022-11-23T02:06:52.0764431Z 2022-11-23T02:06:52.0764693Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0764791Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0764797Z 2022-11-23T02:06:52.0764933Z OK (skipped=1) 2022-11-23T02:06:52.0764939Z 2022-11-23T02:06:52.0765054Z Generating XML reports... 2022-11-23T02:06:52.0765501Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020502.xml 2022-11-23T02:06:52.0765817Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0766195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0766347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0766732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0766911Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0766917Z 2022-11-23T02:06:52.0767012Z Running tests... 2022-11-23T02:06:52.0767279Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0767580Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s) 2022-11-23T02:06:52.0767587Z 2022-11-23T02:06:52.0767850Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0767949Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0767955Z 2022-11-23T02:06:52.0768050Z OK (skipped=1) 2022-11-23T02:06:52.0768055Z 2022-11-23T02:06:52.0768166Z Generating XML reports... 2022-11-23T02:06:52.0768605Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020506.xml 2022-11-23T02:06:52.0768921Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0769296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0769459Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0769848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0770025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0770031Z 2022-11-23T02:06:52.0770125Z Running tests... 2022-11-23T02:06:52.0770392Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0770853Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:06:52.0770860Z 2022-11-23T02:06:52.0771126Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0771222Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0771227Z 2022-11-23T02:06:52.0771324Z OK (skipped=1) 2022-11-23T02:06:52.0771329Z 2022-11-23T02:06:52.0771441Z Generating XML reports... 2022-11-23T02:06:52.0771954Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020510.xml 2022-11-23T02:06:52.0772269Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0772648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0772813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0773195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0773370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0773376Z 2022-11-23T02:06:52.0773467Z Running tests... 2022-11-23T02:06:52.0773733Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0774033Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33886 2022-11-23T02:06:52.0774294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33887 2022-11-23T02:06:52.0774549Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0774929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0775091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0775472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0775647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0775878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0776251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0776423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0776808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0776983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0777207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0777612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0778010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0778224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0778430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0778713Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0778986Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0779072Z ok (5.519s) 2022-11-23T02:06:52.0779078Z 2022-11-23T02:06:52.0779340Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0779436Z Ran 1 test in 5.519s 2022-11-23T02:06:52.0779442Z 2022-11-23T02:06:52.0779519Z OK 2022-11-23T02:06:52.0779524Z 2022-11-23T02:06:52.0779637Z Generating XML reports... 2022-11-23T02:06:52.0780077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020514.xml 2022-11-23T02:06:52.0780390Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0780766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0780980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0781371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0781548Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0781554Z 2022-11-23T02:06:52.0781649Z Running tests... 2022-11-23T02:06:52.0781918Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0782242Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34096 2022-11-23T02:06:52.0782450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34097 2022-11-23T02:06:52.0782709Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0783086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0783296Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0783672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0783847Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0784074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0784448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0784614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0785002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0785178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0785410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0785812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0786209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0786423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0786639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0786973Z STAGE:2022-11-23 02:05:26 34097:34097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0787304Z STAGE:2022-11-23 02:05:26 34096:34096 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0787579Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0787859Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0788200Z STAGE:2022-11-23 02:05:26 34096:34096 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0788554Z STAGE:2022-11-23 02:05:26 34096:34096 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0788777Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0789115Z STAGE:2022-11-23 02:05:27 34097:34097 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0789464Z STAGE:2022-11-23 02:05:27 34097:34097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0789685Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 2 (function operator()) 2022-11-23T02:06:52.0789762Z ok (5.510s) 2022-11-23T02:06:52.0789780Z 2022-11-23T02:06:52.0790038Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0790183Z Ran 1 test in 5.510s 2022-11-23T02:06:52.0790190Z 2022-11-23T02:06:52.0790267Z OK 2022-11-23T02:06:52.0790273Z 2022-11-23T02:06:52.0790384Z Generating XML reports... 2022-11-23T02:06:52.0790830Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020523.xml 2022-11-23T02:06:52.0791143Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0791516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0791681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0792070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0792248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0792301Z 2022-11-23T02:06:52.0792394Z Running tests... 2022-11-23T02:06:52.0792667Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0792990Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34312 2022-11-23T02:06:52.0793198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34313 2022-11-23T02:06:52.0793455Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0793829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0793990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0794375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0794560Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0794790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0795164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0795327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0795701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0795877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0796101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0796502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0796901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0797120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0797334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0797671Z STAGE:2022-11-23 02:05:36 34313:34313 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0798006Z STAGE:2022-11-23 02:05:36 34312:34312 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:06:52.0798279Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0798553Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0798889Z STAGE:2022-11-23 02:05:36 34312:34312 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0799240Z STAGE:2022-11-23 02:05:36 34312:34312 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0799534Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0799874Z STAGE:2022-11-23 02:05:36 34313:34313 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:06:52.0800223Z STAGE:2022-11-23 02:05:36 34313:34313 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:06:52.0800446Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:06:52.0800531Z ok (5.411s) 2022-11-23T02:06:52.0800538Z 2022-11-23T02:06:52.0800804Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0800900Z Ran 1 test in 5.411s 2022-11-23T02:06:52.0800906Z 2022-11-23T02:06:52.0800982Z OK 2022-11-23T02:06:52.0800988Z 2022-11-23T02:06:52.0801096Z Generating XML reports... 2022-11-23T02:06:52.0801533Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020533.xml 2022-11-23T02:06:52.0801898Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0802272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0802435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0802824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0803004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0803010Z 2022-11-23T02:06:52.0803102Z Running tests... 2022-11-23T02:06:52.0803367Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0803641Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s) 2022-11-23T02:06:52.0803650Z 2022-11-23T02:06:52.0803919Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0804015Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0804021Z 2022-11-23T02:06:52.0804112Z OK (skipped=1) 2022-11-23T02:06:52.0804117Z 2022-11-23T02:06:52.0804225Z Generating XML reports... 2022-11-23T02:06:52.0804669Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020542.xml 2022-11-23T02:06:52.0804983Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0805360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0805521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0805907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0806090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0806098Z 2022-11-23T02:06:52.0806193Z Running tests... 2022-11-23T02:06:52.0806460Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0806719Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:06:52.0806725Z 2022-11-23T02:06:52.0806988Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0807075Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0807093Z 2022-11-23T02:06:52.0807176Z OK (skipped=1) 2022-11-23T02:06:52.0807181Z 2022-11-23T02:06:52.0807291Z Generating XML reports... 2022-11-23T02:06:52.0807736Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020546.xml 2022-11-23T02:06:52.0808051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0808487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0808653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0809044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0809220Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0809226Z 2022-11-23T02:06:52.0809321Z Running tests... 2022-11-23T02:06:52.0809587Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0809880Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:06:52.0809886Z 2022-11-23T02:06:52.0810150Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0810247Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0810299Z 2022-11-23T02:06:52.0810393Z OK (skipped=1) 2022-11-23T02:06:52.0810402Z 2022-11-23T02:06:52.0810514Z Generating XML reports... 2022-11-23T02:06:52.0811022Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020550.xml 2022-11-23T02:06:52.0811337Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0811711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0811873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0812254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0812429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0812435Z 2022-11-23T02:06:52.0812530Z Running tests... 2022-11-23T02:06:52.0812783Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0813076Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s) 2022-11-23T02:06:52.0813083Z 2022-11-23T02:06:52.0813345Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0813441Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0813447Z 2022-11-23T02:06:52.0813539Z OK (skipped=1) 2022-11-23T02:06:52.0813544Z 2022-11-23T02:06:52.0813651Z Generating XML reports... 2022-11-23T02:06:52.0814090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020554.xml 2022-11-23T02:06:52.0814403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0814776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0814937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0815328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0815504Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0815509Z 2022-11-23T02:06:52.0815601Z Running tests... 2022-11-23T02:06:52.0815868Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0816138Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.001s) 2022-11-23T02:06:52.0816144Z 2022-11-23T02:06:52.0816405Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0816504Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0816510Z 2022-11-23T02:06:52.0816601Z OK (skipped=1) 2022-11-23T02:06:52.0816606Z 2022-11-23T02:06:52.0816717Z Generating XML reports... 2022-11-23T02:06:52.0817218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020559.xml 2022-11-23T02:06:52.0817541Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0817920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0818083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0818467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0818643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0818649Z 2022-11-23T02:06:52.0818743Z Running tests... 2022-11-23T02:06:52.0819005Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0819284Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.001s) 2022-11-23T02:06:52.0819339Z 2022-11-23T02:06:52.0819609Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0819696Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0819710Z 2022-11-23T02:06:52.0819792Z OK (skipped=1) 2022-11-23T02:06:52.0819798Z 2022-11-23T02:06:52.0819917Z Generating XML reports... 2022-11-23T02:06:52.0820361Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020603.xml 2022-11-23T02:06:52.0820675Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0821049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0821211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0821598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0821782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0821788Z 2022-11-23T02:06:52.0821880Z Running tests... 2022-11-23T02:06:52.0822144Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0822456Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34924 2022-11-23T02:06:52.0822661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34925 2022-11-23T02:06:52.0822918Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0823295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0823458Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0823847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0824030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0824255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0824628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0824790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0825174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0825340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0825565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0825965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0826421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0826645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0826857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0827140Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0827415Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0827503Z ok (7.336s) 2022-11-23T02:06:52.0827509Z 2022-11-23T02:06:52.0827774Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0827870Z Ran 1 test in 7.336s 2022-11-23T02:06:52.0827876Z 2022-11-23T02:06:52.0827954Z OK 2022-11-23T02:06:52.0827959Z 2022-11-23T02:06:52.0828068Z Generating XML reports... 2022-11-23T02:06:52.0828566Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020607.xml 2022-11-23T02:06:52.0828880Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0829253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0829414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0829801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0829976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0829982Z 2022-11-23T02:06:52.0830075Z Running tests... 2022-11-23T02:06:52.0830338Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0830606Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s) 2022-11-23T02:06:52.0830620Z 2022-11-23T02:06:52.0830882Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0830968Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0830974Z 2022-11-23T02:06:52.0831065Z OK (skipped=1) 2022-11-23T02:06:52.0831071Z 2022-11-23T02:06:52.0831179Z Generating XML reports... 2022-11-23T02:06:52.0831624Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020618.xml 2022-11-23T02:06:52.0831946Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0832319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0832479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0832867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0833051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0833057Z 2022-11-23T02:06:52.0833149Z Running tests... 2022-11-23T02:06:52.0833411Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0833715Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35210 2022-11-23T02:06:52.0833926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35211 2022-11-23T02:06:52.0834186Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0834560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0834721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0835164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0835342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0835574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0835951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0836116Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0836503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0836671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0836897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0837305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0837755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0837970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0838187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0838427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynb8ns3a 2022-11-23T02:06:52.0838681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynb8ns3a/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0838959Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0839197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2l4nqb4r 2022-11-23T02:06:52.0839449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2l4nqb4r/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0839735Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0839821Z ok (5.418s) 2022-11-23T02:06:52.0839827Z 2022-11-23T02:06:52.0840096Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0840191Z Ran 1 test in 5.419s 2022-11-23T02:06:52.0840197Z 2022-11-23T02:06:52.0840274Z OK 2022-11-23T02:06:52.0840280Z 2022-11-23T02:06:52.0840390Z Generating XML reports... 2022-11-23T02:06:52.0840834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020622.xml 2022-11-23T02:06:52.0841148Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0841522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0841688Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0842077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0842257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0842263Z 2022-11-23T02:06:52.0842358Z Running tests... 2022-11-23T02:06:52.0842622Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0842964Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35420 2022-11-23T02:06:52.0843171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35421 2022-11-23T02:06:52.0843426Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:06:52.0843800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0844017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0844409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0844588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0844814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:06:52.0845189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0845349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0845732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0845908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0846138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:06:52.0846586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0846983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:06:52.0847195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:06:52.0847409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:06:52.0847647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmc55jxp 2022-11-23T02:06:52.0847898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmc55jxp/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0848165Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0848406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0iru_2p0 2022-11-23T02:06:52.0848661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0iru_2p0/_remote_module_non_scriptable.py 2022-11-23T02:06:52.0848939Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:06:52.0849710Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:06:52.0850488Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:06:52.0850625Z ok (7.425s) 2022-11-23T02:06:52.0850631Z 2022-11-23T02:06:52.0850901Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0851000Z Ran 1 test in 7.425s 2022-11-23T02:06:52.0851007Z 2022-11-23T02:06:52.0851074Z OK 2022-11-23T02:06:52.0851089Z 2022-11-23T02:06:52.0851189Z Generating XML reports... 2022-11-23T02:06:52.0851631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020632.xml 2022-11-23T02:06:52.0851951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0852390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0852554Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0852947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0853124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0853130Z 2022-11-23T02:06:52.0853227Z Running tests... 2022-11-23T02:06:52.0853490Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0853962Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s) 2022-11-23T02:06:52.0853969Z 2022-11-23T02:06:52.0854234Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0854382Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0854392Z 2022-11-23T02:06:52.0854484Z OK (skipped=1) 2022-11-23T02:06:52.0854489Z 2022-11-23T02:06:52.0854602Z Generating XML reports... 2022-11-23T02:06:52.0855051Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020643.xml 2022-11-23T02:06:52.0855370Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:06:52.0855748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:06:52.0855915Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:06:52.0856306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:06:52.0856488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:06:52.0856493Z 2022-11-23T02:06:52.0856594Z Running tests... 2022-11-23T02:06:52.0856855Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0857325Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:06:52.0857346Z 2022-11-23T02:06:52.0857598Z ---------------------------------------------------------------------- 2022-11-23T02:06:52.0857697Z Ran 1 test in 0.002s 2022-11-23T02:06:52.0857703Z 2022-11-23T02:06:52.0857796Z OK (skipped=1) 2022-11-23T02:06:52.0857801Z 2022-11-23T02:06:52.0857914Z Generating XML reports... 2022-11-23T02:06:52.0858365Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020647.xml 2022-11-23T02:06:52.0858371Z 2022-11-23T02:06:52.0858865Z ##[endgroup] 2022-11-23T02:06:52.0859349Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_dw02y2dq) 2022-11-23T02:06:52.0859362Z 2022-11-23T02:06:52.0859566Z Running distributed tests for the nccl backend with file init_method in shard 2 of 2 2022-11-23T02:06:52.0860094Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:06:51.881685] 2022-11-23T02:36:07.8320430Z 2022-11-23T02:36:07.8321512Z Expand the folded group to see the log file of distributed/test_distributed_spawn 2022-11-23T02:36:07.8325240Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_7dv6wrqt) 2022-11-23T02:36:07.8327368Z 2022-11-23T02:36:07.8433280Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-11-23T02:36:07.8490953Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8495361Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8495961Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8496661Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8497365Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8498092Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8498995Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8499773Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8500569Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8501327Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8502177Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8503000Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8504034Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8504865Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8505650Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8506428Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8507092Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8507802Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8508525Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8509253Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8510026Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8510903Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8511596Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8512257Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8512957Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8513660Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8514361Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8515054Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8515728Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8516292Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8516970Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8517648Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8518290Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8518965Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8519676Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8520355Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8520942Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8521632Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8522339Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8522992Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8523750Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8524456Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8525105Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8525846Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8526606Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8527294Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8528018Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8528720Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8529401Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8530024Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8531162Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8531876Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8532667Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8533384Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8534100Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8534686Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8535367Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8536124Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8536798Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8537450Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8538111Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8538652Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8539168Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8540227Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8541093Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8542343Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8543646Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8544901Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8545546Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8597466Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8649945Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8651782Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8653400Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8654796Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8656204Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8657348Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8658797Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8660353Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8661654Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8663565Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8665050Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8666375Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8668006Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8669406Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8670954Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8672629Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8674474Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8676068Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8677449Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8678988Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8680877Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8682677Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8684544Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8686261Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8687848Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8689497Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8691101Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8692265Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8693311Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8694315Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8695222Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8696196Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8696986Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8697929Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8698820Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8699860Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8700804Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8701926Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8702863Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8703878Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8704769Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8705718Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8706749Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8707823Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8708752Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8709777Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8710783Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8711784Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8712857Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8713887Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8714798Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8715552Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8716364Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8717278Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8718148Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8719163Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8720330Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8721468Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8722680Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8723717Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8724773Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8725929Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8726969Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8728088Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8729079Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8730067Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8731997Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8733080Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8734029Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8735034Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8736117Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8737010Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8738014Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8738989Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8740254Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8741263Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8742457Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8743568Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8745022Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8746488Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8747871Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8749266Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8750544Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8752057Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8753459Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8754854Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8756153Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8757213Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8758277Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8759229Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8760238Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8761233Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8762207Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8763071Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8764049Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8765027Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8766023Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8766925Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8767889Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8768894Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8769912Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8771174Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8772287Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8773325Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8774449Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8775400Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8776273Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8777145Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8778193Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8779207Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8780188Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8781379Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8782531Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8783498Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8784510Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8785476Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8786516Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8787591Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8788603Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8789611Z test_gather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8790629Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8843236Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8843757Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8844200Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8844691Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8845117Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8845494Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8845942Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8846510Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8847016Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8847529Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8847998Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8848355Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8848742Z test_isend (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8849184Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8849690Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8850189Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8850832Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8851710Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8852339Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8852874Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8853497Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8854097Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8854708Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8855466Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8856025Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8856682Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8857290Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8857856Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8858432Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8859053Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8859683Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8860475Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8861139Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8861883Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8862527Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8863181Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8863814Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8864369Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8865015Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8865707Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8866372Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8867009Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8867603Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8868176Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8868586Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8869072Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8869588Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8870134Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8870600Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8871094Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8871493Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8871870Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8872234Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8872670Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8873094Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8873595Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8874179Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8874597Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8875066Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8875568Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8876053Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8876523Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8876921Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8877312Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8877805Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8878353Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8878861Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8879359Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8879851Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8880321Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8880806Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8881330Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8881891Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8882383Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8882863Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8883256Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8883756Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8884205Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8884731Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8885266Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8885739Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8886243Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8886662Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8887148Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8887655Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8888156Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8888633Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8889150Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8889609Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.8890515Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.8891397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.8891952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.8892581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.8893122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.8893382Z 2022-11-23T02:36:07.8893510Z Running tests... 2022-11-23T02:36:07.8893934Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8894570Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35838 2022-11-23T02:36:07.8895228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35839 2022-11-23T02:36:07.8895872Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.8896621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.8897204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.8897924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.8898508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.8899073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.8899837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.8900450Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.8901093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.8901600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.8902085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.8902801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.8903574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.8904188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.8904748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.8905387Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:36:07.8906358Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:36:07.8907152Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.8907755Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:36:07.8908722Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:36:07.8909510Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.8910044Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:36:07.8910963Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:36:07.8911715Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:36:07.8912596Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:36:07.8913315Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:36:07.8914270Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:36:07.8915032Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:36:07.8915991Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:36:07.8916743Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:36:07.8917665Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:36:07.8918308Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-11-23T02:36:07.8919216Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-11-23T02:36:07.8919820Z ok (5.722s) 2022-11-23T02:36:07.8919991Z 2022-11-23T02:36:07.8920285Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8920616Z Ran 1 test in 5.723s 2022-11-23T02:36:07.8920808Z 2022-11-23T02:36:07.8920946Z OK 2022-11-23T02:36:07.8921122Z 2022-11-23T02:36:07.8921290Z Generating XML reports... 2022-11-23T02:36:07.8921972Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020655.xml 2022-11-23T02:36:07.8922678Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.8923415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.8923951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.8924656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.8925333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.8925609Z 2022-11-23T02:36:07.8925760Z Running tests... 2022-11-23T02:36:07.8926300Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8926842Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s) 2022-11-23T02:36:07.8927200Z 2022-11-23T02:36:07.8927536Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8928042Z Ran 1 test in 0.004s 2022-11-23T02:36:07.8928246Z 2022-11-23T02:36:07.8928400Z OK (skipped=1) 2022-11-23T02:36:07.8928604Z 2022-11-23T02:36:07.8928771Z Generating XML reports... 2022-11-23T02:36:07.8929455Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020706.xml 2022-11-23T02:36:07.8930257Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.8931274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.8931772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.8932484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.8933113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.8933394Z 2022-11-23T02:36:07.8933521Z Running tests... 2022-11-23T02:36:07.8934049Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8934665Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36118 2022-11-23T02:36:07.8935199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36119 2022-11-23T02:36:07.8935750Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.8936545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.8937127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.8938358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.8939304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.8940145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.8941390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.8942302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.8943673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.8944771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.8945830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.8947199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.8948638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.8949696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.8950640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.8951481Z ok (4.923s) 2022-11-23T02:36:07.8952028Z 2022-11-23T02:36:07.8952706Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8953543Z Ran 1 test in 4.923s 2022-11-23T02:36:07.8953946Z 2022-11-23T02:36:07.8954210Z OK 2022-11-23T02:36:07.8954574Z 2022-11-23T02:36:07.8954905Z Generating XML reports... 2022-11-23T02:36:07.8956324Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020710.xml 2022-11-23T02:36:07.8957648Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.8958890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.8959896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.8961191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.8962323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.8962816Z 2022-11-23T02:36:07.8963101Z Running tests... 2022-11-23T02:36:07.8964041Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8966578Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.655s) 2022-11-23T02:36:07.8967872Z 2022-11-23T02:36:07.8968501Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8969351Z Ran 1 test in 0.655s 2022-11-23T02:36:07.8969618Z 2022-11-23T02:36:07.8969905Z OK (skipped=1) 2022-11-23T02:36:07.8970300Z 2022-11-23T02:36:07.8970704Z Generating XML reports... 2022-11-23T02:36:07.8972099Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020719.xml 2022-11-23T02:36:07.8973643Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.8975103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.8976146Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.8977524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.8978513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.8979002Z 2022-11-23T02:36:07.8979197Z Running tests... 2022-11-23T02:36:07.8980157Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8981149Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.001s) 2022-11-23T02:36:07.8981828Z 2022-11-23T02:36:07.8982564Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8983261Z Ran 1 test in 0.002s 2022-11-23T02:36:07.8983619Z 2022-11-23T02:36:07.8983866Z OK (skipped=1) 2022-11-23T02:36:07.8984255Z 2022-11-23T02:36:07.8984450Z Generating XML reports... 2022-11-23T02:36:07.8985777Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020724.xml 2022-11-23T02:36:07.8987262Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.8988684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.8989756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.8990928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.8991983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.8992415Z 2022-11-23T02:36:07.8992578Z Running tests... 2022-11-23T02:36:07.8993582Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8994817Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.001s) 2022-11-23T02:36:07.8995571Z 2022-11-23T02:36:07.8996228Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.8996952Z Ran 1 test in 0.002s 2022-11-23T02:36:07.8997356Z 2022-11-23T02:36:07.8997658Z OK (skipped=1) 2022-11-23T02:36:07.8998001Z 2022-11-23T02:36:07.8998227Z Generating XML reports... 2022-11-23T02:36:07.8999453Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020728.xml 2022-11-23T02:36:07.9000941Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9002418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9003490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9004934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9006045Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9006606Z 2022-11-23T02:36:07.9006895Z Running tests... 2022-11-23T02:36:07.9007903Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9009069Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36519 2022-11-23T02:36:07.9010397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36520 2022-11-23T02:36:07.9011951Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9013481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9014653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9015941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9016992Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9017782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9018545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9019001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9019754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9020339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9020858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9021592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9022413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9022924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9023402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9023996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9mgzsyg 2022-11-23T02:36:07.9024727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9mgzsyg/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9025323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuw604fp_ 2022-11-23T02:36:07.9025883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuw604fp_/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9026603Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9027184Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9028282Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9028978Z warnings.warn( 2022-11-23T02:36:07.9029387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9030314Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9030858Z warnings.warn( 2022-11-23T02:36:07.9031233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9031718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9032182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9032667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9033148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9033709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9034258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9034846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9035397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9035948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9036540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9036934Z ok (7.827s) 2022-11-23T02:36:07.9037112Z 2022-11-23T02:36:07.9037404Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9037806Z Ran 1 test in 7.827s 2022-11-23T02:36:07.9038003Z 2022-11-23T02:36:07.9038126Z OK 2022-11-23T02:36:07.9038292Z 2022-11-23T02:36:07.9038438Z Generating XML reports... 2022-11-23T02:36:07.9039146Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020732.xml 2022-11-23T02:36:07.9040054Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9040852Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9041441Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9042181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9042742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9043024Z 2022-11-23T02:36:07.9043196Z Running tests... 2022-11-23T02:36:07.9043675Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9044395Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36739 2022-11-23T02:36:07.9045282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36740 2022-11-23T02:36:07.9045894Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9046627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9047121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9047725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9048237Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9048726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9049440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9049932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9050589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9051146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9051708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9052518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9053352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9053926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9054551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9055250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4rvg8c8p 2022-11-23T02:36:07.9055907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4rvg8c8p/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9056543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qq8hi55 2022-11-23T02:36:07.9057102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qq8hi55/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9057761Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9058295Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9058835Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9059505Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9060654Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9061354Z warnings.warn( 2022-11-23T02:36:07.9061909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9062922Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9063510Z warnings.warn( 2022-11-23T02:36:07.9063901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9064454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9064942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9065373Z ok (5.832s) 2022-11-23T02:36:07.9065516Z 2022-11-23T02:36:07.9065804Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9066136Z Ran 1 test in 5.832s 2022-11-23T02:36:07.9066293Z 2022-11-23T02:36:07.9066378Z OK 2022-11-23T02:36:07.9066488Z 2022-11-23T02:36:07.9066608Z Generating XML reports... 2022-11-23T02:36:07.9067239Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020744.xml 2022-11-23T02:36:07.9067910Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9068561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9069060Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9069691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9070226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9070488Z 2022-11-23T02:36:07.9070573Z Running tests... 2022-11-23T02:36:07.9071072Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9071712Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36957 2022-11-23T02:36:07.9072353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36958 2022-11-23T02:36:07.9072962Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9073766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9074256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9074875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9075333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9075894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9076677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9077328Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9078023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9078537Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9079132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9079941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9080798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9081453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9082084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9082773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ryyu_zz 2022-11-23T02:36:07.9083407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ryyu_zz/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9084028Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9084644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmnjq3m9 2022-11-23T02:36:07.9085320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmnjq3m9/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9086026Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9087044Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9087715Z warnings.warn( 2022-11-23T02:36:07.9088229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9089256Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9089866Z warnings.warn( 2022-11-23T02:36:07.9090401Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9091103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9091639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9092238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9092844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9093394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9093974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9094337Z ok (6.023s) 2022-11-23T02:36:07.9094461Z 2022-11-23T02:36:07.9094760Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9095176Z Ran 1 test in 6.023s 2022-11-23T02:36:07.9095385Z 2022-11-23T02:36:07.9095530Z OK 2022-11-23T02:36:07.9095783Z 2022-11-23T02:36:07.9095969Z Generating XML reports... 2022-11-23T02:36:07.9096728Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020754.xml 2022-11-23T02:36:07.9097436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9098064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9098524Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9099170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9099706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9099977Z 2022-11-23T02:36:07.9100133Z Running tests... 2022-11-23T02:36:07.9100592Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9101286Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37175 2022-11-23T02:36:07.9101921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37176 2022-11-23T02:36:07.9102418Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9103104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9103557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9104253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9104758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9105324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9106083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9106633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9107252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9107781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9108236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9108946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9109727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9110281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9110905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9111440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlnt8g9h 2022-11-23T02:36:07.9112034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlnt8g9h/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9112730Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9113299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9s2eu0j 2022-11-23T02:36:07.9113879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9s2eu0j/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9114535Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9115543Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9116266Z warnings.warn( 2022-11-23T02:36:07.9116709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9117705Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9118388Z warnings.warn( 2022-11-23T02:36:07.9118862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9119328Z ok (5.718s) 2022-11-23T02:36:07.9119522Z 2022-11-23T02:36:07.9119878Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9120270Z Ran 1 test in 5.719s 2022-11-23T02:36:07.9120481Z 2022-11-23T02:36:07.9120551Z OK 2022-11-23T02:36:07.9120805Z 2022-11-23T02:36:07.9120962Z Generating XML reports... 2022-11-23T02:36:07.9121685Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020804.xml 2022-11-23T02:36:07.9122487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9123248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9123797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9124495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9125022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9125278Z 2022-11-23T02:36:07.9125416Z Running tests... 2022-11-23T02:36:07.9125970Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9126740Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37393 2022-11-23T02:36:07.9127420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37394 2022-11-23T02:36:07.9127966Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9128740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9129269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9129859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9130335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9130925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9131585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9132034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9132759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9133290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9133786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9134587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9135387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9135998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9136598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9137210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gtvrfob 2022-11-23T02:36:07.9137856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gtvrfob/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9138510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb51_3rtr 2022-11-23T02:36:07.9139080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb51_3rtr/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9139777Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9140475Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9141531Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9142092Z warnings.warn( 2022-11-23T02:36:07.9142467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9143378Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9143971Z warnings.warn( 2022-11-23T02:36:07.9144380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9144924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9145435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9145861Z ok (7.740s) 2022-11-23T02:36:07.9146040Z 2022-11-23T02:36:07.9146395Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9146843Z Ran 1 test in 7.741s 2022-11-23T02:36:07.9147031Z 2022-11-23T02:36:07.9147120Z OK 2022-11-23T02:36:07.9147230Z 2022-11-23T02:36:07.9147394Z Generating XML reports... 2022-11-23T02:36:07.9148197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020814.xml 2022-11-23T02:36:07.9148930Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9149690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9150269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9151007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9151599Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9151810Z 2022-11-23T02:36:07.9151969Z Running tests... 2022-11-23T02:36:07.9152502Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9153203Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37613 2022-11-23T02:36:07.9153899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37614 2022-11-23T02:36:07.9154468Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9155243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9155789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9156502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9157031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9157572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9158261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9158806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9159497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9160002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9160526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9161253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9161974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9162498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9162974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9163474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcw2c7at 2022-11-23T02:36:07.9164008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcw2c7at/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9164583Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9165038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2f_57759 2022-11-23T02:36:07.9165566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2f_57759/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9166256Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9167137Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9167691Z warnings.warn( 2022-11-23T02:36:07.9168073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9168979Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9169517Z warnings.warn( 2022-11-23T02:36:07.9169873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9170362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9171097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9171462Z ok (7.646s) 2022-11-23T02:36:07.9171605Z 2022-11-23T02:36:07.9171898Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9172226Z Ran 1 test in 7.647s 2022-11-23T02:36:07.9172383Z 2022-11-23T02:36:07.9172469Z OK 2022-11-23T02:36:07.9172578Z 2022-11-23T02:36:07.9172698Z Generating XML reports... 2022-11-23T02:36:07.9173327Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020826.xml 2022-11-23T02:36:07.9174051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9174754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9175313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9176016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9176545Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9176795Z 2022-11-23T02:36:07.9176880Z Running tests... 2022-11-23T02:36:07.9177387Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9178087Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37833 2022-11-23T02:36:07.9178800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37834 2022-11-23T02:36:07.9179377Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9180215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9180755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9181442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9181900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9182423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9183167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9183670Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9184273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9184748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9185344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9186131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9186957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9187588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9188160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9188718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqeiom8p 2022-11-23T02:36:07.9189412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqeiom8p/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9190106Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9190647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqkysxfk 2022-11-23T02:36:07.9191210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqkysxfk/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9191886Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9192478Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9193134Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9194167Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9194807Z warnings.warn( 2022-11-23T02:36:07.9195278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9196295Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:07.9196864Z warnings.warn( 2022-11-23T02:36:07.9197331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9197976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9198569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9198988Z ok (5.626s) 2022-11-23T02:36:07.9199176Z 2022-11-23T02:36:07.9199490Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9199900Z Ran 1 test in 5.626s 2022-11-23T02:36:07.9200174Z 2022-11-23T02:36:07.9200243Z OK 2022-11-23T02:36:07.9200461Z 2022-11-23T02:36:07.9200621Z Generating XML reports... 2022-11-23T02:36:07.9201313Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020837.xml 2022-11-23T02:36:07.9202087Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9202825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9203373Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9204092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9204597Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9204867Z 2022-11-23T02:36:07.9205014Z Running tests... 2022-11-23T02:36:07.9205535Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9206944Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.588s) 2022-11-23T02:36:07.9207634Z 2022-11-23T02:36:07.9207961Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9208382Z Ran 1 test in 0.589s 2022-11-23T02:36:07.9208587Z 2022-11-23T02:36:07.9208717Z OK (skipped=1) 2022-11-23T02:36:07.9208870Z 2022-11-23T02:36:07.9208990Z Generating XML reports... 2022-11-23T02:36:07.9209699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020847.xml 2022-11-23T02:36:07.9210401Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9211154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9211712Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9212414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9212986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9213341Z 2022-11-23T02:36:07.9213488Z Running tests... 2022-11-23T02:36:07.9214014Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9214615Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38117 2022-11-23T02:36:07.9215227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38118 2022-11-23T02:36:07.9215748Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9216429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9216938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9217571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9218153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9218613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9219278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9219729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9220408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9220945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9221396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9222082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9222792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9223333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9223887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9224291Z ok (5.534s) 2022-11-23T02:36:07.9224460Z 2022-11-23T02:36:07.9224876Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9225305Z Ran 1 test in 5.535s 2022-11-23T02:36:07.9225509Z 2022-11-23T02:36:07.9225640Z OK 2022-11-23T02:36:07.9225813Z 2022-11-23T02:36:07.9225955Z Generating XML reports... 2022-11-23T02:36:07.9226567Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020852.xml 2022-11-23T02:36:07.9227229Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9227966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9228516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9229240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9229805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9230064Z 2022-11-23T02:36:07.9230196Z Running tests... 2022-11-23T02:36:07.9230652Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9232056Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.605s) 2022-11-23T02:36:07.9232756Z 2022-11-23T02:36:07.9233083Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9233513Z Ran 1 test in 0.605s 2022-11-23T02:36:07.9233714Z 2022-11-23T02:36:07.9233871Z OK (skipped=1) 2022-11-23T02:36:07.9234043Z 2022-11-23T02:36:07.9234207Z Generating XML reports... 2022-11-23T02:36:07.9234936Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020902.xml 2022-11-23T02:36:07.9235703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9236407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9236907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9237589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9238134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9238407Z 2022-11-23T02:36:07.9238559Z Running tests... 2022-11-23T02:36:07.9239061Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9239686Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38386 2022-11-23T02:36:07.9240376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38387 2022-11-23T02:36:07.9240912Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9241695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9242300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9243013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9243580Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9244189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9244934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9245569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9246215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9246782Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9247303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9248066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9248876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9249505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9250054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9250808Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9252271Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9252737Z ok (5.223s) 2022-11-23T02:36:07.9252930Z 2022-11-23T02:36:07.9253271Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9253656Z Ran 1 test in 5.223s 2022-11-23T02:36:07.9253861Z 2022-11-23T02:36:07.9254060Z OK 2022-11-23T02:36:07.9254249Z 2022-11-23T02:36:07.9254418Z Generating XML reports... 2022-11-23T02:36:07.9255077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020906.xml 2022-11-23T02:36:07.9255774Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9256503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9257130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9257795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9258381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9258634Z 2022-11-23T02:36:07.9258762Z Running tests... 2022-11-23T02:36:07.9259221Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9259834Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s) 2022-11-23T02:36:07.9260172Z 2022-11-23T02:36:07.9260518Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9260942Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9261147Z 2022-11-23T02:36:07.9261296Z OK (skipped=1) 2022-11-23T02:36:07.9261494Z 2022-11-23T02:36:07.9261661Z Generating XML reports... 2022-11-23T02:36:07.9262479Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020916.xml 2022-11-23T02:36:07.9263206Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9263933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9264398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9265053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9265622Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9265916Z 2022-11-23T02:36:07.9266142Z Running tests... 2022-11-23T02:36:07.9266665Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9267118Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.9267781Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38662 2022-11-23T02:36:07.9268453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38663 2022-11-23T02:36:07.9268992Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9269771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9270321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9271005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9271555Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9272043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9272814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9273343Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9273981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9274525Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9275050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9275843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9276588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9277187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9277723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9278300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi976m071 2022-11-23T02:36:07.9278942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi976m071/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9279616Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9280246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0b26nso 2022-11-23T02:36:07.9280872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0b26nso/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9281461Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9281939Z ok (8.227s) 2022-11-23T02:36:07.9282106Z 2022-11-23T02:36:07.9282416Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9282851Z Ran 1 test in 8.227s 2022-11-23T02:36:07.9283063Z 2022-11-23T02:36:07.9283214Z OK 2022-11-23T02:36:07.9283394Z 2022-11-23T02:36:07.9283576Z Generating XML reports... 2022-11-23T02:36:07.9284238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020920.xml 2022-11-23T02:36:07.9285009Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9285710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9286220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9286877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9287401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9287696Z 2022-11-23T02:36:07.9287843Z Running tests... 2022-11-23T02:36:07.9288454Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9288939Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.9289598Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38882 2022-11-23T02:36:07.9290255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38883 2022-11-23T02:36:07.9290878Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9291670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9292275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9292998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9293575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9294078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9294785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9295292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9295952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9296536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9297079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9297862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9298615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9299244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9299818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9300411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9n53y1q4 2022-11-23T02:36:07.9300991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9n53y1q4/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9301667Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9302249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e3y6qpi 2022-11-23T02:36:07.9302810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e3y6qpi/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9303521Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9304046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9304626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9305120Z ok (7.776s) 2022-11-23T02:36:07.9305266Z 2022-11-23T02:36:07.9305611Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9305942Z Ran 1 test in 7.777s 2022-11-23T02:36:07.9306149Z 2022-11-23T02:36:07.9306219Z OK 2022-11-23T02:36:07.9306348Z 2022-11-23T02:36:07.9306466Z Generating XML reports... 2022-11-23T02:36:07.9307258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020932.xml 2022-11-23T02:36:07.9307990Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9308695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9309228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9309910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9310420Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9310695Z 2022-11-23T02:36:07.9310848Z Running tests... 2022-11-23T02:36:07.9311379Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9311895Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.9312601Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39102 2022-11-23T02:36:07.9313258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39103 2022-11-23T02:36:07.9313876Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9314610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9315098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9315705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9316286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9316843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9317621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9318425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9318937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9319644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9320175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9320675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9321519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9322096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9322685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9323277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedjokqvr 2022-11-23T02:36:07.9323971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedjokqvr/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9324547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnn1u3slu 2022-11-23T02:36:07.9325192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnn1u3slu/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9325834Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9326409Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9326918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9327405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9327759Z ok (7.527s) 2022-11-23T02:36:07.9327883Z 2022-11-23T02:36:07.9328165Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9328555Z Ran 1 test in 7.528s 2022-11-23T02:36:07.9328771Z 2022-11-23T02:36:07.9328861Z OK 2022-11-23T02:36:07.9328991Z 2022-11-23T02:36:07.9329110Z Generating XML reports... 2022-11-23T02:36:07.9329747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020944.xml 2022-11-23T02:36:07.9330414Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9331496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9332026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9332715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9333194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9333420Z 2022-11-23T02:36:07.9333522Z Running tests... 2022-11-23T02:36:07.9333958Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9334403Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-11-23T02:36:07.9334959Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39322 2022-11-23T02:36:07.9335480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39323 2022-11-23T02:36:07.9335968Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9336625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9337061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9337640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9338106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9338595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9339220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9339725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9340321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9340787Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9341291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9342022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9342810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9343340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9343798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9344297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1i8noj_ 2022-11-23T02:36:07.9344830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1i8noj_/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9345411Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9345889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijq15326 2022-11-23T02:36:07.9346406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijq15326/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9346967Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9347338Z ok (7.623s) 2022-11-23T02:36:07.9347480Z 2022-11-23T02:36:07.9347752Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9348077Z Ran 1 test in 7.624s 2022-11-23T02:36:07.9348236Z 2022-11-23T02:36:07.9348313Z OK 2022-11-23T02:36:07.9348439Z 2022-11-23T02:36:07.9348552Z Generating XML reports... 2022-11-23T02:36:07.9349175Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020956.xml 2022-11-23T02:36:07.9349842Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9350460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9350894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9351482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9351954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9352173Z 2022-11-23T02:36:07.9352265Z Running tests... 2022-11-23T02:36:07.9352671Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9353135Z test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:07.9353409Z 2022-11-23T02:36:07.9353663Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9353984Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9354137Z 2022-11-23T02:36:07.9354235Z OK (skipped=1) 2022-11-23T02:36:07.9354377Z 2022-11-23T02:36:07.9354492Z Generating XML reports... 2022-11-23T02:36:07.9355103Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021008.xml 2022-11-23T02:36:07.9355761Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9356397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9356873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9357463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9357927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9358148Z 2022-11-23T02:36:07.9358245Z Running tests... 2022-11-23T02:36:07.9358657Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9359162Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s) 2022-11-23T02:36:07.9359477Z 2022-11-23T02:36:07.9359748Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9360060Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9360213Z 2022-11-23T02:36:07.9360376Z OK (skipped=1) 2022-11-23T02:36:07.9360528Z 2022-11-23T02:36:07.9360636Z Generating XML reports... 2022-11-23T02:36:07.9361245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021012.xml 2022-11-23T02:36:07.9361901Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9362530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9363026Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9363682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9364130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9364381Z 2022-11-23T02:36:07.9364521Z Running tests... 2022-11-23T02:36:07.9365036Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9365572Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:36:07.9365918Z 2022-11-23T02:36:07.9366185Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9366529Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9366535Z 2022-11-23T02:36:07.9366657Z OK (skipped=1) 2022-11-23T02:36:07.9366663Z 2022-11-23T02:36:07.9366804Z Generating XML reports... 2022-11-23T02:36:07.9367284Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021016.xml 2022-11-23T02:36:07.9367634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9368004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9368210Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9368600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9368816Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9368822Z 2022-11-23T02:36:07.9369010Z Running tests... 2022-11-23T02:36:07.9369277Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9369556Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s) 2022-11-23T02:36:07.9369563Z 2022-11-23T02:36:07.9369822Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9370014Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9370020Z 2022-11-23T02:36:07.9370141Z OK (skipped=1) 2022-11-23T02:36:07.9370147Z 2022-11-23T02:36:07.9370257Z Generating XML reports... 2022-11-23T02:36:07.9370888Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021020.xml 2022-11-23T02:36:07.9371219Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9371595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9371757Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9372176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9372385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9372391Z 2022-11-23T02:36:07.9372518Z Running tests... 2022-11-23T02:36:07.9372819Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9373208Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.001s) 2022-11-23T02:36:07.9373218Z 2022-11-23T02:36:07.9373534Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9373631Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9373637Z 2022-11-23T02:36:07.9373733Z OK (skipped=1) 2022-11-23T02:36:07.9373739Z 2022-11-23T02:36:07.9373838Z Generating XML reports... 2022-11-23T02:36:07.9374283Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021024.xml 2022-11-23T02:36:07.9374598Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9374975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9375180Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9375601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9375876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9375883Z 2022-11-23T02:36:07.9376006Z Running tests... 2022-11-23T02:36:07.9376311Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9376625Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s) 2022-11-23T02:36:07.9376632Z 2022-11-23T02:36:07.9376945Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9377072Z Ran 1 test in 0.003s 2022-11-23T02:36:07.9377078Z 2022-11-23T02:36:07.9377200Z OK (skipped=1) 2022-11-23T02:36:07.9377205Z 2022-11-23T02:36:07.9377342Z Generating XML reports... 2022-11-23T02:36:07.9377820Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021029.xml 2022-11-23T02:36:07.9378145Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9378519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9378715Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9379134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9379342Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9379348Z 2022-11-23T02:36:07.9379445Z Running tests... 2022-11-23T02:36:07.9379742Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9379988Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:07.9380033Z 2022-11-23T02:36:07.9380292Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9380396Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9380404Z 2022-11-23T02:36:07.9380530Z OK (skipped=1) 2022-11-23T02:36:07.9380535Z 2022-11-23T02:36:07.9380643Z Generating XML reports... 2022-11-23T02:36:07.9381122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021033.xml 2022-11-23T02:36:07.9381438Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9381845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9382038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9382425Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9382637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9382643Z 2022-11-23T02:36:07.9382740Z Running tests... 2022-11-23T02:36:07.9383114Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9383457Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40004 2022-11-23T02:36:07.9383665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40005 2022-11-23T02:36:07.9383922Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9384308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9384503Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9384924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9385131Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9385450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9385869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9386035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9386411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9386620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9386872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9387309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9387710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9387963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9388210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9388588Z STAGE:2022-11-23 02:10:40 40005:40005 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9388976Z STAGE:2022-11-23 02:10:40 40004:40004 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9389290Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9389601Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9389977Z STAGE:2022-11-23 02:10:40 40004:40004 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9390350Z STAGE:2022-11-23 02:10:40 40005:40005 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9390729Z STAGE:2022-11-23 02:10:40 40004:40004 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9391090Z STAGE:2022-11-23 02:10:40 40005:40005 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9391313Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9391538Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9391879Z STAGE:2022-11-23 02:10:40 40004:40004 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9392215Z STAGE:2022-11-23 02:10:40 40004:40004 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9392575Z STAGE:2022-11-23 02:10:40 40004:40004 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9392910Z STAGE:2022-11-23 02:10:40 40005:40005 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9393317Z STAGE:2022-11-23 02:10:40 40005:40005 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9393686Z STAGE:2022-11-23 02:10:40 40005:40005 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9393764Z ok (5.518s) 2022-11-23T02:36:07.9393779Z 2022-11-23T02:36:07.9394035Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9394137Z Ran 1 test in 5.518s 2022-11-23T02:36:07.9394143Z 2022-11-23T02:36:07.9394226Z OK 2022-11-23T02:36:07.9394232Z 2022-11-23T02:36:07.9394339Z Generating XML reports... 2022-11-23T02:36:07.9394793Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021037.xml 2022-11-23T02:36:07.9395113Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9395488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9395710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9396101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9396285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9396291Z 2022-11-23T02:36:07.9396382Z Running tests... 2022-11-23T02:36:07.9396653Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9396973Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40220 2022-11-23T02:36:07.9397181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40221 2022-11-23T02:36:07.9397438Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9397825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9397990Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9398386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9398567Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9398797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9399170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9399321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9399756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9399932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9400165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9400571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9401005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9401230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9401452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9401790Z STAGE:2022-11-23 02:10:50 40220:40220 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9402073Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9402468Z STAGE:2022-11-23 02:10:50 40221:40221 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9402765Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9403109Z STAGE:2022-11-23 02:10:50 40220:40220 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9403454Z STAGE:2022-11-23 02:10:50 40221:40221 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9403807Z STAGE:2022-11-23 02:10:50 40220:40220 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9404377Z STAGE:2022-11-23 02:10:50 40221:40221 ActivityProfilerController.cpp:310] Completed Stage: Post Processing[W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9404383Z 2022-11-23T02:36:07.9404611Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9404949Z STAGE:2022-11-23 02:10:50 40221:40221 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9405348Z STAGE:2022-11-23 02:10:50 40221:40221 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9405710Z STAGE:2022-11-23 02:10:50 40221:40221 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9406045Z STAGE:2022-11-23 02:10:50 40220:40220 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9406386Z STAGE:2022-11-23 02:10:50 40220:40220 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9406737Z STAGE:2022-11-23 02:10:50 40220:40220 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9406823Z ok (5.523s) 2022-11-23T02:36:07.9406829Z 2022-11-23T02:36:07.9407101Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9407188Z Ran 1 test in 5.524s 2022-11-23T02:36:07.9407204Z 2022-11-23T02:36:07.9407271Z OK 2022-11-23T02:36:07.9407280Z 2022-11-23T02:36:07.9407393Z Generating XML reports... 2022-11-23T02:36:07.9407846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021047.xml 2022-11-23T02:36:07.9408167Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9408547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9408709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9409098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9409279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9409284Z 2022-11-23T02:36:07.9409378Z Running tests... 2022-11-23T02:36:07.9409650Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9409918Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:07.9409924Z 2022-11-23T02:36:07.9410193Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9410293Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9410299Z 2022-11-23T02:36:07.9410398Z OK (skipped=1) 2022-11-23T02:36:07.9410403Z 2022-11-23T02:36:07.9410586Z Generating XML reports... 2022-11-23T02:36:07.9411335Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021056.xml 2022-11-23T02:36:07.9411884Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9412275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9412442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9412912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9413103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9413109Z 2022-11-23T02:36:07.9413208Z Running tests... 2022-11-23T02:36:07.9413466Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9413717Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:07.9413723Z 2022-11-23T02:36:07.9413991Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9414092Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9414098Z 2022-11-23T02:36:07.9414195Z OK (skipped=1) 2022-11-23T02:36:07.9414200Z 2022-11-23T02:36:07.9414313Z Generating XML reports... 2022-11-23T02:36:07.9414764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021100.xml 2022-11-23T02:36:07.9415138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9415515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9415683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9416072Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9416254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9416260Z 2022-11-23T02:36:07.9416353Z Running tests... 2022-11-23T02:36:07.9416621Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9416949Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40568 2022-11-23T02:36:07.9417164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40569 2022-11-23T02:36:07.9417430Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9417813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9417975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9418364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9418540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9418773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9419137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9419305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9419696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9419877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9420108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9420508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9420913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9421136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9421357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9421700Z STAGE:2022-11-23 02:11:08 40568:40568 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9422029Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9422378Z STAGE:2022-11-23 02:11:08 40569:40569 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9422658Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9423216Z STAGE:2022-11-23 02:11:08 40569:40569 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:11:08 40568:40568 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9423223Z 2022-11-23T02:36:07.9423573Z STAGE:2022-11-23 02:11:08 40569:40569 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9423924Z STAGE:2022-11-23 02:11:08 40568:40568 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9424161Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9424434Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9424530Z ok (5.710s) 2022-11-23T02:36:07.9424536Z 2022-11-23T02:36:07.9424808Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9424911Z Ran 1 test in 5.710s 2022-11-23T02:36:07.9424917Z 2022-11-23T02:36:07.9424999Z OK 2022-11-23T02:36:07.9425005Z 2022-11-23T02:36:07.9425113Z Generating XML reports... 2022-11-23T02:36:07.9425563Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021105.xml 2022-11-23T02:36:07.9425868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9426248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9426411Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9426801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9426984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9426990Z 2022-11-23T02:36:07.9427088Z Running tests... 2022-11-23T02:36:07.9427352Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9427679Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40784 2022-11-23T02:36:07.9427886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40785 2022-11-23T02:36:07.9428145Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9428527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9428700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9429095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9429270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9429500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9429878Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9430043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9430429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9430616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9430986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9431397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9431797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9432013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9432216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9432551Z STAGE:2022-11-23 02:11:18 40784:40784 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9432825Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9433166Z STAGE:2022-11-23 02:11:18 40785:40785 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9433447Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9433838Z STAGE:2022-11-23 02:11:18 40784:40784 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9434183Z STAGE:2022-11-23 02:11:18 40785:40785 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9434544Z STAGE:2022-11-23 02:11:18 40784:40784 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9434899Z STAGE:2022-11-23 02:11:18 40785:40785 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9435125Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9435352Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9435442Z ok (5.321s) 2022-11-23T02:36:07.9435448Z 2022-11-23T02:36:07.9435713Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9435818Z Ran 1 test in 5.321s 2022-11-23T02:36:07.9435827Z 2022-11-23T02:36:07.9435903Z OK 2022-11-23T02:36:07.9435909Z 2022-11-23T02:36:07.9436017Z Generating XML reports... 2022-11-23T02:36:07.9436462Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021115.xml 2022-11-23T02:36:07.9436782Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9437164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9437333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9437727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9437910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9437916Z 2022-11-23T02:36:07.9438000Z Running tests... 2022-11-23T02:36:07.9438273Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9439200Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.589s) 2022-11-23T02:36:07.9439224Z 2022-11-23T02:36:07.9439477Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9439580Z Ran 1 test in 0.589s 2022-11-23T02:36:07.9439586Z 2022-11-23T02:36:07.9439683Z OK (skipped=1) 2022-11-23T02:36:07.9439689Z 2022-11-23T02:36:07.9439803Z Generating XML reports... 2022-11-23T02:36:07.9440246Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021125.xml 2022-11-23T02:36:07.9440618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9441007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9441173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9441568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9441750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9441756Z 2022-11-23T02:36:07.9441848Z Running tests... 2022-11-23T02:36:07.9442117Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9443056Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.582s) 2022-11-23T02:36:07.9443108Z 2022-11-23T02:36:07.9443375Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9443479Z Ran 1 test in 0.582s 2022-11-23T02:36:07.9443485Z 2022-11-23T02:36:07.9443581Z OK (skipped=1) 2022-11-23T02:36:07.9443587Z 2022-11-23T02:36:07.9443701Z Generating XML reports... 2022-11-23T02:36:07.9444150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021129.xml 2022-11-23T02:36:07.9444469Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9444850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9445012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9445412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9445596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9445602Z 2022-11-23T02:36:07.9445703Z Running tests... 2022-11-23T02:36:07.9445977Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9446291Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41132 2022-11-23T02:36:07.9446498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41133 2022-11-23T02:36:07.9446752Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9447139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9447310Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9447709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9447889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9448115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9448490Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9448651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9449035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9449210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9449436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9449900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9450308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9450661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9450887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9451175Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9451457Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9451554Z ok (5.724s) 2022-11-23T02:36:07.9451560Z 2022-11-23T02:36:07.9451831Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9451937Z Ran 1 test in 5.725s 2022-11-23T02:36:07.9452004Z 2022-11-23T02:36:07.9452073Z OK 2022-11-23T02:36:07.9452100Z 2022-11-23T02:36:07.9452202Z Generating XML reports... 2022-11-23T02:36:07.9452656Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021134.xml 2022-11-23T02:36:07.9452980Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9453366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9453536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9453929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9454114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9454119Z 2022-11-23T02:36:07.9454222Z Running tests... 2022-11-23T02:36:07.9454500Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9454827Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41342 2022-11-23T02:36:07.9455045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41343 2022-11-23T02:36:07.9455308Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9455694Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9455865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9456260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9456445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9456682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9457069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9457240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9457631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9457813Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9458046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9458436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9458842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9459125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9459351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9459581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:36:07.9459809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:36:07.9460222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:07.9460628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:07.9460910Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9461191Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9461425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:36:07.9461884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:36:07.9462119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:36:07.9462526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:36:07.9462756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T02:36:07.9462985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T02:36:07.9463384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:36:07.9463786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T02:36:07.9463885Z ok (6.117s) 2022-11-23T02:36:07.9463892Z 2022-11-23T02:36:07.9464165Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9464269Z Ran 1 test in 6.118s 2022-11-23T02:36:07.9464275Z 2022-11-23T02:36:07.9464365Z OK 2022-11-23T02:36:07.9464370Z 2022-11-23T02:36:07.9464492Z Generating XML reports... 2022-11-23T02:36:07.9464949Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021144.xml 2022-11-23T02:36:07.9465254Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9465638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9465808Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9466206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9466396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9466402Z 2022-11-23T02:36:07.9466503Z Running tests... 2022-11-23T02:36:07.9466778Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9467088Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41587 2022-11-23T02:36:07.9467304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41588 2022-11-23T02:36:07.9467569Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9467955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9468125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9468607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9468798Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9469034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9469428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9469601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9469997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9470182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9470416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9470883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9471286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9471509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9471716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9472003Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9472286Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9472380Z ok (5.432s) 2022-11-23T02:36:07.9472385Z 2022-11-23T02:36:07.9472656Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9472757Z Ran 1 test in 5.433s 2022-11-23T02:36:07.9472763Z 2022-11-23T02:36:07.9472849Z OK 2022-11-23T02:36:07.9472859Z 2022-11-23T02:36:07.9472980Z Generating XML reports... 2022-11-23T02:36:07.9473435Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021155.xml 2022-11-23T02:36:07.9473758Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9474139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9474303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9474695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9474876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9474881Z 2022-11-23T02:36:07.9474979Z Running tests... 2022-11-23T02:36:07.9475252Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9475705Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9475711Z 2022-11-23T02:36:07.9475981Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9476084Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9476089Z 2022-11-23T02:36:07.9476190Z OK (skipped=1) 2022-11-23T02:36:07.9476195Z 2022-11-23T02:36:07.9476314Z Generating XML reports... 2022-11-23T02:36:07.9476748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021204.xml 2022-11-23T02:36:07.9477077Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9477461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9477632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9478084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9478273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9478279Z 2022-11-23T02:36:07.9478383Z Running tests... 2022-11-23T02:36:07.9478657Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9479089Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9479095Z 2022-11-23T02:36:07.9479363Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9479470Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9479475Z 2022-11-23T02:36:07.9479575Z OK (skipped=1) 2022-11-23T02:36:07.9479580Z 2022-11-23T02:36:07.9479701Z Generating XML reports... 2022-11-23T02:36:07.9480149Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021209.xml 2022-11-23T02:36:07.9480525Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9480905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9481069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9481462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9481649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9481655Z 2022-11-23T02:36:07.9481756Z Running tests... 2022-11-23T02:36:07.9482028Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9482476Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9482484Z 2022-11-23T02:36:07.9482760Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9482867Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9482873Z 2022-11-23T02:36:07.9482956Z OK (skipped=1) 2022-11-23T02:36:07.9482979Z 2022-11-23T02:36:07.9483079Z Generating XML reports... 2022-11-23T02:36:07.9483530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021213.xml 2022-11-23T02:36:07.9483852Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9484236Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9484407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9484798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9484982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9484993Z 2022-11-23T02:36:07.9485098Z Running tests... 2022-11-23T02:36:07.9485372Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9485810Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9485816Z 2022-11-23T02:36:07.9486086Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9486190Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9486196Z 2022-11-23T02:36:07.9486294Z OK (skipped=1) 2022-11-23T02:36:07.9486299Z 2022-11-23T02:36:07.9486412Z Generating XML reports... 2022-11-23T02:36:07.9486860Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021217.xml 2022-11-23T02:36:07.9487183Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9487617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9487788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9488187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9488367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9488373Z 2022-11-23T02:36:07.9488474Z Running tests... 2022-11-23T02:36:07.9488729Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9489160Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9489184Z 2022-11-23T02:36:07.9489437Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9489541Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9489598Z 2022-11-23T02:36:07.9489701Z OK (skipped=1) 2022-11-23T02:36:07.9489709Z 2022-11-23T02:36:07.9489821Z Generating XML reports... 2022-11-23T02:36:07.9490266Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021221.xml 2022-11-23T02:36:07.9490637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9491013Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9491188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9491578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9491759Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9491764Z 2022-11-23T02:36:07.9491867Z Running tests... 2022-11-23T02:36:07.9492140Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9492573Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9492580Z 2022-11-23T02:36:07.9492853Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9492958Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9492963Z 2022-11-23T02:36:07.9493062Z OK (skipped=1) 2022-11-23T02:36:07.9493067Z 2022-11-23T02:36:07.9493184Z Generating XML reports... 2022-11-23T02:36:07.9493635Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021226.xml 2022-11-23T02:36:07.9493960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9494343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9494515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9494916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9495083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9495107Z 2022-11-23T02:36:07.9495191Z Running tests... 2022-11-23T02:36:07.9495467Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9495907Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9495913Z 2022-11-23T02:36:07.9496183Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9496285Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9496291Z 2022-11-23T02:36:07.9496392Z OK (skipped=1) 2022-11-23T02:36:07.9496398Z 2022-11-23T02:36:07.9496513Z Generating XML reports... 2022-11-23T02:36:07.9497027Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021230.xml 2022-11-23T02:36:07.9497361Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9497738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9497911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9498307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9498492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9498497Z 2022-11-23T02:36:07.9498597Z Running tests... 2022-11-23T02:36:07.9498869Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9499301Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9499381Z 2022-11-23T02:36:07.9499662Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9499767Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9499772Z 2022-11-23T02:36:07.9499872Z OK (skipped=1) 2022-11-23T02:36:07.9499877Z 2022-11-23T02:36:07.9499994Z Generating XML reports... 2022-11-23T02:36:07.9500452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021234.xml 2022-11-23T02:36:07.9500772Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9501141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9501309Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9501706Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9501896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9501901Z 2022-11-23T02:36:07.9502001Z Running tests... 2022-11-23T02:36:07.9502279Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9502692Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9502699Z 2022-11-23T02:36:07.9502970Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9503072Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9503078Z 2022-11-23T02:36:07.9503178Z OK (skipped=1) 2022-11-23T02:36:07.9503184Z 2022-11-23T02:36:07.9503300Z Generating XML reports... 2022-11-23T02:36:07.9503751Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021239.xml 2022-11-23T02:36:07.9504073Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9504462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9504631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9505025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9505203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9505209Z 2022-11-23T02:36:07.9505310Z Running tests... 2022-11-23T02:36:07.9505583Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9505889Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9505895Z 2022-11-23T02:36:07.9506167Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9506277Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9506283Z 2022-11-23T02:36:07.9506442Z OK (skipped=1) 2022-11-23T02:36:07.9506449Z 2022-11-23T02:36:07.9506550Z Generating XML reports... 2022-11-23T02:36:07.9507005Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021243.xml 2022-11-23T02:36:07.9507327Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9507710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9507882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9508278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9508463Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9508469Z 2022-11-23T02:36:07.9508571Z Running tests... 2022-11-23T02:36:07.9508903Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9509318Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9509324Z 2022-11-23T02:36:07.9509595Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9509698Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9509704Z 2022-11-23T02:36:07.9509805Z OK (skipped=1) 2022-11-23T02:36:07.9509810Z 2022-11-23T02:36:07.9509930Z Generating XML reports... 2022-11-23T02:36:07.9510381Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021247.xml 2022-11-23T02:36:07.9510695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9511084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9511262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9511658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9511842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9511848Z 2022-11-23T02:36:07.9511945Z Running tests... 2022-11-23T02:36:07.9512217Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9512626Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9512652Z 2022-11-23T02:36:07.9512908Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9513016Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9513021Z 2022-11-23T02:36:07.9513121Z OK (skipped=1) 2022-11-23T02:36:07.9513127Z 2022-11-23T02:36:07.9513246Z Generating XML reports... 2022-11-23T02:36:07.9513702Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021251.xml 2022-11-23T02:36:07.9514024Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9514411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9514581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9514975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9515159Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9515165Z 2022-11-23T02:36:07.9515263Z Running tests... 2022-11-23T02:36:07.9515539Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9516001Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:07.9516011Z 2022-11-23T02:36:07.9516288Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9516392Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9516397Z 2022-11-23T02:36:07.9516500Z OK (skipped=1) 2022-11-23T02:36:07.9516505Z 2022-11-23T02:36:07.9516619Z Generating XML reports... 2022-11-23T02:36:07.9517068Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021255.xml 2022-11-23T02:36:07.9517390Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9517776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9517939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9518335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9518575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9518581Z 2022-11-23T02:36:07.9518666Z Running tests... 2022-11-23T02:36:07.9518942Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9519231Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9519237Z 2022-11-23T02:36:07.9519506Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9519603Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9519608Z 2022-11-23T02:36:07.9519713Z OK (skipped=1) 2022-11-23T02:36:07.9519718Z 2022-11-23T02:36:07.9519833Z Generating XML reports... 2022-11-23T02:36:07.9520283Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021300.xml 2022-11-23T02:36:07.9520609Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9520999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9521171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9521566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9521751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9521757Z 2022-11-23T02:36:07.9521860Z Running tests... 2022-11-23T02:36:07.9522136Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9522404Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9522410Z 2022-11-23T02:36:07.9522684Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9522792Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9522800Z 2022-11-23T02:36:07.9522899Z OK (skipped=1) 2022-11-23T02:36:07.9522904Z 2022-11-23T02:36:07.9523022Z Generating XML reports... 2022-11-23T02:36:07.9523477Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021304.xml 2022-11-23T02:36:07.9523805Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9524192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9524347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9524742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9524924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9524929Z 2022-11-23T02:36:07.9525032Z Running tests... 2022-11-23T02:36:07.9525365Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9525638Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9525644Z 2022-11-23T02:36:07.9525925Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9526029Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9526035Z 2022-11-23T02:36:07.9526135Z OK (skipped=1) 2022-11-23T02:36:07.9526140Z 2022-11-23T02:36:07.9526262Z Generating XML reports... 2022-11-23T02:36:07.9526713Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021308.xml 2022-11-23T02:36:07.9527038Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9527423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9527648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9528050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9528239Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9528245Z 2022-11-23T02:36:07.9528350Z Running tests... 2022-11-23T02:36:07.9528622Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9528895Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9528901Z 2022-11-23T02:36:07.9529173Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9529279Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9529285Z 2022-11-23T02:36:07.9529387Z OK (skipped=1) 2022-11-23T02:36:07.9529392Z 2022-11-23T02:36:07.9529509Z Generating XML reports... 2022-11-23T02:36:07.9529948Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021312.xml 2022-11-23T02:36:07.9530274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9530730Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9530900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9531296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9531482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9531487Z 2022-11-23T02:36:07.9531587Z Running tests... 2022-11-23T02:36:07.9531862Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9532130Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9532142Z 2022-11-23T02:36:07.9532417Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9532522Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9532528Z 2022-11-23T02:36:07.9532624Z OK (skipped=1) 2022-11-23T02:36:07.9532629Z 2022-11-23T02:36:07.9532748Z Generating XML reports... 2022-11-23T02:36:07.9533201Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021317.xml 2022-11-23T02:36:07.9533524Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9533908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9534078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9534471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9534724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9534730Z 2022-11-23T02:36:07.9534835Z Running tests... 2022-11-23T02:36:07.9535113Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9535376Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9535383Z 2022-11-23T02:36:07.9535654Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9535742Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9535747Z 2022-11-23T02:36:07.9535847Z OK (skipped=1) 2022-11-23T02:36:07.9535852Z 2022-11-23T02:36:07.9535968Z Generating XML reports... 2022-11-23T02:36:07.9536421Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021321.xml 2022-11-23T02:36:07.9536746Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9537194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9537366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9537761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9537945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9537951Z 2022-11-23T02:36:07.9538042Z Running tests... 2022-11-23T02:36:07.9538314Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9538576Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9538582Z 2022-11-23T02:36:07.9538855Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9538962Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9538968Z 2022-11-23T02:36:07.9539068Z OK (skipped=1) 2022-11-23T02:36:07.9539073Z 2022-11-23T02:36:07.9539186Z Generating XML reports... 2022-11-23T02:36:07.9539637Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021325.xml 2022-11-23T02:36:07.9539959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9540343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9540513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9540910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9541096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9541103Z 2022-11-23T02:36:07.9541186Z Running tests... 2022-11-23T02:36:07.9541463Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9541735Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9541741Z 2022-11-23T02:36:07.9542012Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9542116Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9542122Z 2022-11-23T02:36:07.9542223Z OK (skipped=1) 2022-11-23T02:36:07.9542229Z 2022-11-23T02:36:07.9542351Z Generating XML reports... 2022-11-23T02:36:07.9542800Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021329.xml 2022-11-23T02:36:07.9543124Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9543508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9543738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9544147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9544329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9544335Z 2022-11-23T02:36:07.9544438Z Running tests... 2022-11-23T02:36:07.9544712Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9544974Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9544980Z 2022-11-23T02:36:07.9545256Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9545360Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9545365Z 2022-11-23T02:36:07.9545460Z OK (skipped=1) 2022-11-23T02:36:07.9545465Z 2022-11-23T02:36:07.9545584Z Generating XML reports... 2022-11-23T02:36:07.9546035Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021333.xml 2022-11-23T02:36:07.9546411Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9546796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9546952Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9547347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9547535Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9547541Z 2022-11-23T02:36:07.9547642Z Running tests... 2022-11-23T02:36:07.9547917Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9548173Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9548182Z 2022-11-23T02:36:07.9548458Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9548563Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9548569Z 2022-11-23T02:36:07.9548668Z OK (skipped=1) 2022-11-23T02:36:07.9548674Z 2022-11-23T02:36:07.9548790Z Generating XML reports... 2022-11-23T02:36:07.9549238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021337.xml 2022-11-23T02:36:07.9549560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9549943Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9550104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9550496Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9550687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9550693Z 2022-11-23T02:36:07.9550794Z Running tests... 2022-11-23T02:36:07.9551062Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9551319Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:07.9551325Z 2022-11-23T02:36:07.9551595Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9551701Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9551707Z 2022-11-23T02:36:07.9551804Z OK (skipped=1) 2022-11-23T02:36:07.9551809Z 2022-11-23T02:36:07.9551911Z Generating XML reports... 2022-11-23T02:36:07.9552361Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021341.xml 2022-11-23T02:36:07.9552684Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9553127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9553301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9553698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9553875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9553881Z 2022-11-23T02:36:07.9553974Z Running tests... 2022-11-23T02:36:07.9554238Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9554503Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s) 2022-11-23T02:36:07.9554509Z 2022-11-23T02:36:07.9554770Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9554867Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9554921Z 2022-11-23T02:36:07.9555015Z OK (skipped=1) 2022-11-23T02:36:07.9555024Z 2022-11-23T02:36:07.9555136Z Generating XML reports... 2022-11-23T02:36:07.9555578Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021345.xml 2022-11-23T02:36:07.9555892Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9556265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9556427Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9556813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9556988Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9556994Z 2022-11-23T02:36:07.9557089Z Running tests... 2022-11-23T02:36:07.9557351Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9557639Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s) 2022-11-23T02:36:07.9557645Z 2022-11-23T02:36:07.9557908Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9557995Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9558000Z 2022-11-23T02:36:07.9558093Z OK (skipped=1) 2022-11-23T02:36:07.9558099Z 2022-11-23T02:36:07.9558208Z Generating XML reports... 2022-11-23T02:36:07.9558652Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021349.xml 2022-11-23T02:36:07.9558967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9559341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9559506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9559897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9560077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9560082Z 2022-11-23T02:36:07.9560175Z Running tests... 2022-11-23T02:36:07.9560437Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9560693Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9560699Z 2022-11-23T02:36:07.9560957Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9561054Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9561060Z 2022-11-23T02:36:07.9561152Z OK (skipped=1) 2022-11-23T02:36:07.9561157Z 2022-11-23T02:36:07.9561268Z Generating XML reports... 2022-11-23T02:36:07.9561769Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021354.xml 2022-11-23T02:36:07.9562092Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9562468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9562633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9563021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9563201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9563207Z 2022-11-23T02:36:07.9563291Z Running tests... 2022-11-23T02:36:07.9563553Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9563866Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43583 2022-11-23T02:36:07.9564125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43584 2022-11-23T02:36:07.9564380Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9564760Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9564922Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9565309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9565486Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9565711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9566086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9566253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9566635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9566812Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9567038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9567440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9567842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9568065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9568278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9568561Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9568835Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9568921Z ok (5.420s) 2022-11-23T02:36:07.9568927Z 2022-11-23T02:36:07.9569179Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9569274Z Ran 1 test in 5.420s 2022-11-23T02:36:07.9569280Z 2022-11-23T02:36:07.9569358Z OK 2022-11-23T02:36:07.9569363Z 2022-11-23T02:36:07.9569470Z Generating XML reports... 2022-11-23T02:36:07.9569913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021359.xml 2022-11-23T02:36:07.9570227Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9570728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9570898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9571567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9571888Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9571901Z 2022-11-23T02:36:07.9572058Z Running tests... 2022-11-23T02:36:07.9572411Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9572659Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9572665Z 2022-11-23T02:36:07.9572928Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9573023Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9573029Z 2022-11-23T02:36:07.9573121Z OK (skipped=1) 2022-11-23T02:36:07.9573127Z 2022-11-23T02:36:07.9573235Z Generating XML reports... 2022-11-23T02:36:07.9573679Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021408.xml 2022-11-23T02:36:07.9574063Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9574438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9574598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9574981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9575148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9575162Z 2022-11-23T02:36:07.9575247Z Running tests... 2022-11-23T02:36:07.9575515Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9575769Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9575778Z 2022-11-23T02:36:07.9576044Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9576139Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9576145Z 2022-11-23T02:36:07.9576236Z OK (skipped=1) 2022-11-23T02:36:07.9576242Z 2022-11-23T02:36:07.9576352Z Generating XML reports... 2022-11-23T02:36:07.9576793Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021412.xml 2022-11-23T02:36:07.9577106Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9577480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9577644Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9578029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9578210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9578216Z 2022-11-23T02:36:07.9578310Z Running tests... 2022-11-23T02:36:07.9578571Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9578829Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:07.9578835Z 2022-11-23T02:36:07.9579097Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9579196Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9579202Z 2022-11-23T02:36:07.9579293Z OK (skipped=1) 2022-11-23T02:36:07.9579299Z 2022-11-23T02:36:07.9579406Z Generating XML reports... 2022-11-23T02:36:07.9579848Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021416.xml 2022-11-23T02:36:07.9580161Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9580584Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9580748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9581135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9581311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9581317Z 2022-11-23T02:36:07.9581410Z Running tests... 2022-11-23T02:36:07.9581673Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9581980Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43991 2022-11-23T02:36:07.9582188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43992 2022-11-23T02:36:07.9582446Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9582868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9583030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9583417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9583596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9583821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9584192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9584354Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9584737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9584922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9585147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9585547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9585946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9586160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9586376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9586707Z STAGE:2022-11-23 02:14:23 43991:43991 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9586983Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9587322Z STAGE:2022-11-23 02:14:23 43992:43992 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9587597Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9587941Z STAGE:2022-11-23 02:14:24 43992:43992 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9588296Z STAGE:2022-11-23 02:14:24 43992:43992 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9588850Z STAGE:2022-11-23 02:14:24 43991:43991 ActivityProfilerController.cpp:306] Completed Stage: Collection[W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9588856Z 2022-11-23T02:36:07.9589210Z STAGE:2022-11-23 02:14:24 43991:43991 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9589433Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9589816Z STAGE:2022-11-23 02:14:24 43991:43991 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9590154Z STAGE:2022-11-23 02:14:24 43992:43992 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9590496Z STAGE:2022-11-23 02:14:24 43992:43992 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9590832Z STAGE:2022-11-23 02:14:24 43991:43991 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9591188Z STAGE:2022-11-23 02:14:24 43992:43992 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9591539Z STAGE:2022-11-23 02:14:24 43991:43991 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9591869Z STAGE:2022-11-23 02:14:24 43991:43991 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9592198Z STAGE:2022-11-23 02:14:24 43992:43992 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9592586Z STAGE:2022-11-23 02:14:24 43991:43991 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9592936Z STAGE:2022-11-23 02:14:24 43991:43991 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9593269Z STAGE:2022-11-23 02:14:24 43992:43992 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9593619Z STAGE:2022-11-23 02:14:24 43992:43992 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9593704Z ok (5.234s) 2022-11-23T02:36:07.9593710Z 2022-11-23T02:36:07.9593981Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9594078Z Ran 1 test in 5.234s 2022-11-23T02:36:07.9594083Z 2022-11-23T02:36:07.9594152Z OK 2022-11-23T02:36:07.9594168Z 2022-11-23T02:36:07.9594270Z Generating XML reports... 2022-11-23T02:36:07.9594716Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021420.xml 2022-11-23T02:36:07.9595031Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9595405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9595569Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9595957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9596137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9596143Z 2022-11-23T02:36:07.9596237Z Running tests... 2022-11-23T02:36:07.9596501Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9596813Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44207 2022-11-23T02:36:07.9597026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44208 2022-11-23T02:36:07.9597281Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9597656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9597819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9598206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9598381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9598609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9598981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9599199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9599588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9599765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9599994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9600386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9600781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9601001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9601217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9601603Z STAGE:2022-11-23 02:14:33 44208:44208 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9601935Z STAGE:2022-11-23 02:14:33 44207:44207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9602210Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9602485Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9602823Z STAGE:2022-11-23 02:14:33 44208:44208 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9603161Z STAGE:2022-11-23 02:14:33 44207:44207 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9603514Z STAGE:2022-11-23 02:14:33 44208:44208 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9603739Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9604095Z STAGE:2022-11-23 02:14:33 44207:44207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9604318Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9604649Z STAGE:2022-11-23 02:14:33 44208:44208 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9604977Z STAGE:2022-11-23 02:14:33 44207:44207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9605316Z STAGE:2022-11-23 02:14:33 44208:44208 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9605664Z STAGE:2022-11-23 02:14:33 44208:44208 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9606001Z STAGE:2022-11-23 02:14:33 44207:44207 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9606348Z STAGE:2022-11-23 02:14:33 44207:44207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9606681Z STAGE:2022-11-23 02:14:33 44208:44208 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9607014Z STAGE:2022-11-23 02:14:33 44207:44207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9607350Z STAGE:2022-11-23 02:14:33 44208:44208 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9607689Z STAGE:2022-11-23 02:14:33 44208:44208 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9608027Z STAGE:2022-11-23 02:14:33 44207:44207 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9608376Z STAGE:2022-11-23 02:14:33 44207:44207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9608461Z ok (5.522s) 2022-11-23T02:36:07.9608467Z 2022-11-23T02:36:07.9608731Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9608830Z Ran 1 test in 5.523s 2022-11-23T02:36:07.9608839Z 2022-11-23T02:36:07.9608919Z OK 2022-11-23T02:36:07.9608979Z 2022-11-23T02:36:07.9609090Z Generating XML reports... 2022-11-23T02:36:07.9609535Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021430.xml 2022-11-23T02:36:07.9609851Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9610226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9610390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9610833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9611013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9611019Z 2022-11-23T02:36:07.9611113Z Running tests... 2022-11-23T02:36:07.9611380Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9611760Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44423 2022-11-23T02:36:07.9611968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44424 2022-11-23T02:36:07.9612225Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9612601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9612763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9613144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9613321Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9613545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9613916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9614077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9614459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9614636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9614859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9615259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9615661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9615885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9616100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9616434Z STAGE:2022-11-23 02:14:43 44424:44424 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9616768Z STAGE:2022-11-23 02:14:43 44423:44423 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9617044Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9617318Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9617655Z STAGE:2022-11-23 02:14:43 44424:44424 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9617992Z STAGE:2022-11-23 02:14:43 44423:44423 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9618348Z STAGE:2022-11-23 02:14:43 44424:44424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9618753Z STAGE:2022-11-23 02:14:43 44423:44423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9618981Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9619202Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9619538Z STAGE:2022-11-23 02:14:43 44423:44423 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9619870Z STAGE:2022-11-23 02:14:43 44424:44424 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9620195Z STAGE:2022-11-23 02:14:43 44423:44423 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9620547Z STAGE:2022-11-23 02:14:43 44423:44423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9620888Z STAGE:2022-11-23 02:14:43 44424:44424 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9621290Z STAGE:2022-11-23 02:14:43 44424:44424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9621620Z STAGE:2022-11-23 02:14:43 44423:44423 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9621952Z STAGE:2022-11-23 02:14:43 44424:44424 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9622291Z STAGE:2022-11-23 02:14:43 44423:44423 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9622874Z STAGE:2022-11-23 02:14:43 44424:44424 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:14:43 44423:44423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9622881Z 2022-11-23T02:36:07.9623234Z STAGE:2022-11-23 02:14:43 44424:44424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9623325Z ok (5.421s) 2022-11-23T02:36:07.9623331Z 2022-11-23T02:36:07.9623598Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9623696Z Ran 1 test in 5.421s 2022-11-23T02:36:07.9623702Z 2022-11-23T02:36:07.9623779Z OK 2022-11-23T02:36:07.9623785Z 2022-11-23T02:36:07.9623895Z Generating XML reports... 2022-11-23T02:36:07.9624339Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021440.xml 2022-11-23T02:36:07.9624655Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9625030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9625193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9625576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9625765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9625771Z 2022-11-23T02:36:07.9625866Z Running tests... 2022-11-23T02:36:07.9626129Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9626359Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:36:07.9626365Z 2022-11-23T02:36:07.9626626Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9626723Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9626729Z 2022-11-23T02:36:07.9626812Z OK (skipped=1) 2022-11-23T02:36:07.9626828Z 2022-11-23T02:36:07.9626929Z Generating XML reports... 2022-11-23T02:36:07.9627373Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021449.xml 2022-11-23T02:36:07.9627687Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9628124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9628289Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9628679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9628854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9628860Z 2022-11-23T02:36:07.9629068Z Running tests... 2022-11-23T02:36:07.9629415Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9629719Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:36:07.9629725Z 2022-11-23T02:36:07.9630051Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9630205Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9630211Z 2022-11-23T02:36:07.9630440Z OK (skipped=1) 2022-11-23T02:36:07.9630447Z 2022-11-23T02:36:07.9630615Z Generating XML reports... 2022-11-23T02:36:07.9631322Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021453.xml 2022-11-23T02:36:07.9631718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9632152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9632446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9632897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9633128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9633134Z 2022-11-23T02:36:07.9633282Z Running tests... 2022-11-23T02:36:07.9633542Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9668576Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44771 2022-11-23T02:36:07.9668827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44772 2022-11-23T02:36:07.9669082Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9669576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9669735Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9670121Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9670303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9670534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9670925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9671088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9671476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9671652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9671876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9672281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9672682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9673090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9673314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9673439Z skip: Test skipped for ROCm (4.929s) 2022-11-23T02:36:07.9673447Z 2022-11-23T02:36:07.9673726Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9673822Z Ran 1 test in 4.929s 2022-11-23T02:36:07.9673829Z 2022-11-23T02:36:07.9673920Z OK (skipped=1) 2022-11-23T02:36:07.9673927Z 2022-11-23T02:36:07.9674039Z Generating XML reports... 2022-11-23T02:36:07.9674481Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021457.xml 2022-11-23T02:36:07.9674802Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9675181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9675425Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9675821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9675999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9676005Z 2022-11-23T02:36:07.9676103Z Running tests... 2022-11-23T02:36:07.9676371Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9676684Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44974 2022-11-23T02:36:07.9676891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44975 2022-11-23T02:36:07.9677148Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9677529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9677694Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9678083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9678259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9678485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9678863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9679024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9679412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9679590Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9679823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9680226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9680616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9680832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9681045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9681169Z skip: Test skipped for ROCm (4.831s) 2022-11-23T02:36:07.9681175Z 2022-11-23T02:36:07.9681444Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9681542Z Ran 1 test in 4.832s 2022-11-23T02:36:07.9681548Z 2022-11-23T02:36:07.9681643Z OK (skipped=1) 2022-11-23T02:36:07.9681651Z 2022-11-23T02:36:07.9681761Z Generating XML reports... 2022-11-23T02:36:07.9682266Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021507.xml 2022-11-23T02:36:07.9682591Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9682968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9683134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9683526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9683706Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9683711Z 2022-11-23T02:36:07.9683804Z Running tests... 2022-11-23T02:36:07.9684074Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9684408Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:36:07.9684414Z 2022-11-23T02:36:07.9684683Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9684779Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9684784Z 2022-11-23T02:36:07.9684876Z OK (skipped=1) 2022-11-23T02:36:07.9684881Z 2022-11-23T02:36:07.9684991Z Generating XML reports... 2022-11-23T02:36:07.9685438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021516.xml 2022-11-23T02:36:07.9685754Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9686124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9686286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9686677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9686857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9686863Z 2022-11-23T02:36:07.9686954Z Running tests... 2022-11-23T02:36:07.9687221Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9687533Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45243 2022-11-23T02:36:07.9687740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45244 2022-11-23T02:36:07.9687997Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9688374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9688536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9688929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9689107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9689334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9689711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9689873Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9690261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9690441Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9690756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9691232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9691637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9691851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9692071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9692185Z skip: Test skipped for ROCm (4.921s) 2022-11-23T02:36:07.9692199Z 2022-11-23T02:36:07.9692455Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9692552Z Ran 1 test in 4.921s 2022-11-23T02:36:07.9692558Z 2022-11-23T02:36:07.9692649Z OK (skipped=1) 2022-11-23T02:36:07.9692655Z 2022-11-23T02:36:07.9692763Z Generating XML reports... 2022-11-23T02:36:07.9693211Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021520.xml 2022-11-23T02:36:07.9693584Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9693961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9694122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9694512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9694691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9694696Z 2022-11-23T02:36:07.9694787Z Running tests... 2022-11-23T02:36:07.9695055Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9695294Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s) 2022-11-23T02:36:07.9695303Z 2022-11-23T02:36:07.9695569Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9695667Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9695673Z 2022-11-23T02:36:07.9695764Z OK (skipped=1) 2022-11-23T02:36:07.9695769Z 2022-11-23T02:36:07.9695877Z Generating XML reports... 2022-11-23T02:36:07.9696323Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021529.xml 2022-11-23T02:36:07.9696640Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9697015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9697178Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9697556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9697732Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9697744Z 2022-11-23T02:36:07.9697838Z Running tests... 2022-11-23T02:36:07.9698108Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9698416Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45512 2022-11-23T02:36:07.9698622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45513 2022-11-23T02:36:07.9698878Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9699254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9699415Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9699799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9700030Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9700261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9700640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9700800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9701186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9701363Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9701589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9701987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9702458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9702674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9702887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9703032Z skip: Skipped due to small world size. (4.919s) 2022-11-23T02:36:07.9703038Z 2022-11-23T02:36:07.9703303Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9703390Z Ran 1 test in 4.920s 2022-11-23T02:36:07.9703405Z 2022-11-23T02:36:07.9703487Z OK (skipped=1) 2022-11-23T02:36:07.9703493Z 2022-11-23T02:36:07.9703602Z Generating XML reports... 2022-11-23T02:36:07.9704053Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021533.xml 2022-11-23T02:36:07.9704374Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9704752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9704913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9705299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9705476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9705482Z 2022-11-23T02:36:07.9705573Z Running tests... 2022-11-23T02:36:07.9705842Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9706115Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:36:07.9706121Z 2022-11-23T02:36:07.9706384Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9706484Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9706492Z 2022-11-23T02:36:07.9706584Z OK (skipped=1) 2022-11-23T02:36:07.9706589Z 2022-11-23T02:36:07.9706700Z Generating XML reports... 2022-11-23T02:36:07.9707144Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021542.xml 2022-11-23T02:36:07.9707462Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9707840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9708002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9708392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9708566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9708572Z 2022-11-23T02:36:07.9708667Z Running tests... 2022-11-23T02:36:07.9708981Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9709269Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:36:07.9709275Z 2022-11-23T02:36:07.9709540Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9709640Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9709646Z 2022-11-23T02:36:07.9709737Z OK (skipped=1) 2022-11-23T02:36:07.9709743Z 2022-11-23T02:36:07.9709855Z Generating XML reports... 2022-11-23T02:36:07.9710298Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021546.xml 2022-11-23T02:36:07.9710615Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9710989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9711203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9711594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9711770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9711776Z 2022-11-23T02:36:07.9711868Z Running tests... 2022-11-23T02:36:07.9712136Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9713089Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.591s) 2022-11-23T02:36:07.9713096Z 2022-11-23T02:36:07.9713365Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9713461Z Ran 1 test in 0.591s 2022-11-23T02:36:07.9713467Z 2022-11-23T02:36:07.9713558Z OK (skipped=1) 2022-11-23T02:36:07.9713564Z 2022-11-23T02:36:07.9713671Z Generating XML reports... 2022-11-23T02:36:07.9714120Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021550.xml 2022-11-23T02:36:07.9714438Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9714814Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9714975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9715363Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9715539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9715547Z 2022-11-23T02:36:07.9715643Z Running tests... 2022-11-23T02:36:07.9715900Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9716235Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45913 2022-11-23T02:36:07.9716443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45914 2022-11-23T02:36:07.9716696Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9717071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9717231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9717616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9717851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9718081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9718458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9718619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9719003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9719180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9719405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9719804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9720252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9720468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9720680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9721021Z STAGE:2022-11-23 02:15:58 45913:45913 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9721300Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9721635Z STAGE:2022-11-23 02:15:58 45914:45914 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:07.9721914Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9722473Z STAGE:2022-11-23 02:15:58 45914:45914 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:15:58 45913:45913 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:07.9722486Z 2022-11-23T02:36:07.9722842Z STAGE:2022-11-23 02:15:58 45914:45914 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9723185Z STAGE:2022-11-23 02:15:58 45913:45913 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:07.9723410Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9723635Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:07.9723720Z ok (5.420s) 2022-11-23T02:36:07.9723726Z 2022-11-23T02:36:07.9723993Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9724088Z Ran 1 test in 5.420s 2022-11-23T02:36:07.9724093Z 2022-11-23T02:36:07.9724169Z OK 2022-11-23T02:36:07.9724175Z 2022-11-23T02:36:07.9724282Z Generating XML reports... 2022-11-23T02:36:07.9724727Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021555.xml 2022-11-23T02:36:07.9725044Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9725423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9725583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9725969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9726146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9726152Z 2022-11-23T02:36:07.9726244Z Running tests... 2022-11-23T02:36:07.9726511Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9726795Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:36:07.9726858Z 2022-11-23T02:36:07.9727127Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9727224Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9727230Z 2022-11-23T02:36:07.9727322Z OK (skipped=1) 2022-11-23T02:36:07.9727327Z 2022-11-23T02:36:07.9727434Z Generating XML reports... 2022-11-23T02:36:07.9727882Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021605.xml 2022-11-23T02:36:07.9728198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9728563Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9728725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9729111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9729350Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9729356Z 2022-11-23T02:36:07.9729446Z Running tests... 2022-11-23T02:36:07.9729716Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9730826Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.589s) 2022-11-23T02:36:07.9730836Z 2022-11-23T02:36:07.9731319Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9731517Z Ran 1 test in 0.590s 2022-11-23T02:36:07.9731525Z 2022-11-23T02:36:07.9731701Z OK (skipped=1) 2022-11-23T02:36:07.9731723Z 2022-11-23T02:36:07.9731870Z Generating XML reports... 2022-11-23T02:36:07.9732324Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021609.xml 2022-11-23T02:36:07.9732640Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9733014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9733176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9733565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9733741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9733747Z 2022-11-23T02:36:07.9733837Z Running tests... 2022-11-23T02:36:07.9734099Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9734372Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:36:07.9734390Z 2022-11-23T02:36:07.9734642Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9734739Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9734744Z 2022-11-23T02:36:07.9734835Z OK (skipped=1) 2022-11-23T02:36:07.9734840Z 2022-11-23T02:36:07.9734947Z Generating XML reports... 2022-11-23T02:36:07.9735386Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021614.xml 2022-11-23T02:36:07.9735699Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9736071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9736231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9736693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9736875Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9736881Z 2022-11-23T02:36:07.9736973Z Running tests... 2022-11-23T02:36:07.9737240Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9737569Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46331 2022-11-23T02:36:07.9737775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46332 2022-11-23T02:36:07.9738028Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9738398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9738556Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9738994Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9739168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9739393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9739767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9739927Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9740301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9740477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9740702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9741106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9741499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9741712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9741922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9742068Z skip: Skipped due to small world size. (4.921s) 2022-11-23T02:36:07.9742074Z 2022-11-23T02:36:07.9742334Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9742429Z Ran 1 test in 4.921s 2022-11-23T02:36:07.9742434Z 2022-11-23T02:36:07.9742524Z OK (skipped=1) 2022-11-23T02:36:07.9742529Z 2022-11-23T02:36:07.9742636Z Generating XML reports... 2022-11-23T02:36:07.9743079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021618.xml 2022-11-23T02:36:07.9743395Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9743767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9743926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9744308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9744483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9744489Z 2022-11-23T02:36:07.9744580Z Running tests... 2022-11-23T02:36:07.9744845Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9745117Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:36:07.9745126Z 2022-11-23T02:36:07.9745438Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9745535Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9745541Z 2022-11-23T02:36:07.9745624Z OK (skipped=1) 2022-11-23T02:36:07.9745638Z 2022-11-23T02:36:07.9745738Z Generating XML reports... 2022-11-23T02:36:07.9746181Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021627.xml 2022-11-23T02:36:07.9746494Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9746870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9747031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9747414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9747648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9747654Z 2022-11-23T02:36:07.9747745Z Running tests... 2022-11-23T02:36:07.9748014Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9748298Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s) 2022-11-23T02:36:07.9748304Z 2022-11-23T02:36:07.9748565Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9748660Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9748665Z 2022-11-23T02:36:07.9748756Z OK (skipped=1) 2022-11-23T02:36:07.9748761Z 2022-11-23T02:36:07.9748869Z Generating XML reports... 2022-11-23T02:36:07.9749311Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021631.xml 2022-11-23T02:36:07.9749624Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9750005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9750163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9750544Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9750720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9750725Z 2022-11-23T02:36:07.9750818Z Running tests... 2022-11-23T02:36:07.9751083Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9751400Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46666 2022-11-23T02:36:07.9751605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46667 2022-11-23T02:36:07.9751864Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9752237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9752397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9752779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9752953Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9753178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9753550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9753711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9754152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9754330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9754554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9754954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9755351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9755567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9755780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9756054Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9756379Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9756465Z ok (5.319s) 2022-11-23T02:36:07.9756471Z 2022-11-23T02:36:07.9756732Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9756831Z Ran 1 test in 5.319s 2022-11-23T02:36:07.9756837Z 2022-11-23T02:36:07.9756905Z OK 2022-11-23T02:36:07.9756920Z 2022-11-23T02:36:07.9757020Z Generating XML reports... 2022-11-23T02:36:07.9757467Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021635.xml 2022-11-23T02:36:07.9757784Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9758157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9758316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9758710Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9758889Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9758894Z 2022-11-23T02:36:07.9758985Z Running tests... 2022-11-23T02:36:07.9759247Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9759584Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46880 2022-11-23T02:36:07.9759791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46881 2022-11-23T02:36:07.9760046Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9760419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9760581Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9760971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9761148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9761372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9761744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9761906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9762289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9762465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9762690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9763138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9763538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9763754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9763967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9764242Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9764516Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9764600Z ok (6.119s) 2022-11-23T02:36:07.9764606Z 2022-11-23T02:36:07.9764870Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9764964Z Ran 1 test in 6.120s 2022-11-23T02:36:07.9764970Z 2022-11-23T02:36:07.9765091Z OK 2022-11-23T02:36:07.9765097Z 2022-11-23T02:36:07.9765208Z Generating XML reports... 2022-11-23T02:36:07.9765653Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021645.xml 2022-11-23T02:36:07.9765965Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9766338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9766498Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9766882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9767057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9767063Z 2022-11-23T02:36:07.9767155Z Running tests... 2022-11-23T02:36:07.9767421Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9767715Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:36:07.9767721Z 2022-11-23T02:36:07.9767983Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9768080Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9768085Z 2022-11-23T02:36:07.9768167Z OK (skipped=1) 2022-11-23T02:36:07.9768180Z 2022-11-23T02:36:07.9768280Z Generating XML reports... 2022-11-23T02:36:07.9768719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021655.xml 2022-11-23T02:36:07.9769032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9769405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9769566Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9769958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9770134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9770140Z 2022-11-23T02:36:07.9770231Z Running tests... 2022-11-23T02:36:07.9770494Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9770993Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47160 2022-11-23T02:36:07.9771202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47161 2022-11-23T02:36:07.9771456Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9771834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9772079Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9772472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9772647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9772872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9773246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9773407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9773791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9773966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9774239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9774643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9775039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9775256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9775479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:36:07.9775692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9775914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:36:07.9776311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:07.9776711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:07.9776986Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9777260Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9777344Z ok (5.508s) 2022-11-23T02:36:07.9777350Z 2022-11-23T02:36:07.9777612Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9777707Z Ran 1 test in 5.509s 2022-11-23T02:36:07.9777713Z 2022-11-23T02:36:07.9777791Z OK 2022-11-23T02:36:07.9777797Z 2022-11-23T02:36:07.9777907Z Generating XML reports... 2022-11-23T02:36:07.9778350Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021659.xml 2022-11-23T02:36:07.9778665Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9779045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9779205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9779587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9779763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9779769Z 2022-11-23T02:36:07.9779862Z Running tests... 2022-11-23T02:36:07.9780118Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9780396Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s) 2022-11-23T02:36:07.9780403Z 2022-11-23T02:36:07.9780664Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9780758Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9780766Z 2022-11-23T02:36:07.9780906Z OK (skipped=1) 2022-11-23T02:36:07.9780912Z 2022-11-23T02:36:07.9781024Z Generating XML reports... 2022-11-23T02:36:07.9781471Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021709.xml 2022-11-23T02:36:07.9781784Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9782155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9782315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9782699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9782873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9782879Z 2022-11-23T02:36:07.9782970Z Running tests... 2022-11-23T02:36:07.9783287Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9783623Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47442 2022-11-23T02:36:07.9783830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47443 2022-11-23T02:36:07.9784086Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9784460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9784620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9785004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9785179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9785408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9785779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9785932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9786312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9786488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9786710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9787109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9787504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9787725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9787940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9788084Z skip: Skipped due to small world size. (4.810s) 2022-11-23T02:36:07.9788089Z 2022-11-23T02:36:07.9788353Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9788448Z Ran 1 test in 4.810s 2022-11-23T02:36:07.9788453Z 2022-11-23T02:36:07.9788544Z OK (skipped=1) 2022-11-23T02:36:07.9788549Z 2022-11-23T02:36:07.9788656Z Generating XML reports... 2022-11-23T02:36:07.9789097Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021713.xml 2022-11-23T02:36:07.9789411Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9789832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9790001Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9790388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9790564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9790569Z 2022-11-23T02:36:07.9790661Z Running tests... 2022-11-23T02:36:07.9790923Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9791230Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47645 2022-11-23T02:36:07.9791438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47646 2022-11-23T02:36:07.9791684Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9792109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9792272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9792654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9792828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9793053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9793424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9793584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9793969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9794149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9794373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9794774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9795167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9795381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9795593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9795868Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9796145Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9796370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:36:07.9796596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:36:07.9796992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:07.9797384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:07.9797469Z ok (5.831s) 2022-11-23T02:36:07.9797475Z 2022-11-23T02:36:07.9797736Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9797823Z Ran 1 test in 5.831s 2022-11-23T02:36:07.9797829Z 2022-11-23T02:36:07.9797908Z OK 2022-11-23T02:36:07.9797913Z 2022-11-23T02:36:07.9798022Z Generating XML reports... 2022-11-23T02:36:07.9798464Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021722.xml 2022-11-23T02:36:07.9798910Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9799286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9799446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9799829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9800005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9800010Z 2022-11-23T02:36:07.9800102Z Running tests... 2022-11-23T02:36:07.9800366Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9800795Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.009s) 2022-11-23T02:36:07.9800801Z 2022-11-23T02:36:07.9801109Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9801208Z Ran 1 test in 0.009s 2022-11-23T02:36:07.9801213Z 2022-11-23T02:36:07.9801305Z OK (skipped=1) 2022-11-23T02:36:07.9801310Z 2022-11-23T02:36:07.9801417Z Generating XML reports... 2022-11-23T02:36:07.9801857Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021732.xml 2022-11-23T02:36:07.9802173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9802548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9802709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9803093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9803269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9803278Z 2022-11-23T02:36:07.9803364Z Running tests... 2022-11-23T02:36:07.9803627Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9804048Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:36:07.9804054Z 2022-11-23T02:36:07.9804314Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9804408Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9804414Z 2022-11-23T02:36:07.9804505Z OK (skipped=1) 2022-11-23T02:36:07.9804510Z 2022-11-23T02:36:07.9804617Z Generating XML reports... 2022-11-23T02:36:07.9805058Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021736.xml 2022-11-23T02:36:07.9805373Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9805748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9805912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9806295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9806470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9806476Z 2022-11-23T02:36:07.9806567Z Running tests... 2022-11-23T02:36:07.9806831Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9807065Z test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.001s) 2022-11-23T02:36:07.9807071Z 2022-11-23T02:36:07.9807332Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9807428Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9807433Z 2022-11-23T02:36:07.9807527Z OK (skipped=1) 2022-11-23T02:36:07.9807535Z 2022-11-23T02:36:07.9807646Z Generating XML reports... 2022-11-23T02:36:07.9808144Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021740.xml 2022-11-23T02:36:07.9808462Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9808836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9808987Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9809369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9809544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9809549Z 2022-11-23T02:36:07.9809640Z Running tests... 2022-11-23T02:36:07.9809904Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9810250Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48066 2022-11-23T02:36:07.9810457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48067 2022-11-23T02:36:07.9810835Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9811459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9811620Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9812007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9812184Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9812412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9812792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9812951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9813332Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9813508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9813733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9814130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9814524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9814736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9814954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9815230Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9815495Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9815580Z ok (6.409s) 2022-11-23T02:36:07.9815586Z 2022-11-23T02:36:07.9815848Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9815943Z Ran 1 test in 6.409s 2022-11-23T02:36:07.9815948Z 2022-11-23T02:36:07.9816025Z OK 2022-11-23T02:36:07.9816030Z 2022-11-23T02:36:07.9816138Z Generating XML reports... 2022-11-23T02:36:07.9816582Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021744.xml 2022-11-23T02:36:07.9816895Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9817352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9817520Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9817905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9818085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9818091Z 2022-11-23T02:36:07.9818183Z Running tests... 2022-11-23T02:36:07.9818446Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9818697Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.001s) 2022-11-23T02:36:07.9818703Z 2022-11-23T02:36:07.9818964Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9819058Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9819064Z 2022-11-23T02:36:07.9819155Z OK (skipped=1) 2022-11-23T02:36:07.9819210Z 2022-11-23T02:36:07.9819322Z Generating XML reports... 2022-11-23T02:36:07.9819764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021754.xml 2022-11-23T02:36:07.9820078Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9820443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9820605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9820991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9821169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9821175Z 2022-11-23T02:36:07.9821267Z Running tests... 2022-11-23T02:36:07.9821531Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9821847Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48342 2022-11-23T02:36:07.9822054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48343 2022-11-23T02:36:07.9822306Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9822678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9822839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9823222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9823396Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9823624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9824002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9824163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9824548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9824722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9824946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9825345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9825743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9825960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9826228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9826367Z skip: Skipped due to small world size. (5.417s) 2022-11-23T02:36:07.9826381Z 2022-11-23T02:36:07.9826639Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9826735Z Ran 1 test in 5.417s 2022-11-23T02:36:07.9826740Z 2022-11-23T02:36:07.9826832Z OK (skipped=1) 2022-11-23T02:36:07.9826837Z 2022-11-23T02:36:07.9826944Z Generating XML reports... 2022-11-23T02:36:07.9827389Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021758.xml 2022-11-23T02:36:07.9827701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9828073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9828287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9828676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9828853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9828859Z 2022-11-23T02:36:07.9828952Z Running tests... 2022-11-23T02:36:07.9829219Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9829462Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.001s) 2022-11-23T02:36:07.9829468Z 2022-11-23T02:36:07.9829731Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9829826Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9829832Z 2022-11-23T02:36:07.9829921Z OK (skipped=1) 2022-11-23T02:36:07.9829926Z 2022-11-23T02:36:07.9830034Z Generating XML reports... 2022-11-23T02:36:07.9830479Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021808.xml 2022-11-23T02:36:07.9830798Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9831172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9831332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9831707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9831886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9831901Z 2022-11-23T02:36:07.9831985Z Running tests... 2022-11-23T02:36:07.9832250Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9832553Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48611 2022-11-23T02:36:07.9832769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48612 2022-11-23T02:36:07.9833024Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9833398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9833558Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9833941Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9834116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9834341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9834711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9834923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9835317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9835491Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9836946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9837358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9837755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9837968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9838183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9838404Z skip: Skipped due to small world size. (5.317s) 2022-11-23T02:36:07.9838410Z 2022-11-23T02:36:07.9838680Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9838778Z Ran 1 test in 5.318s 2022-11-23T02:36:07.9838783Z 2022-11-23T02:36:07.9838865Z OK (skipped=1) 2022-11-23T02:36:07.9838880Z 2022-11-23T02:36:07.9838979Z Generating XML reports... 2022-11-23T02:36:07.9839420Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021812.xml 2022-11-23T02:36:07.9839738Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9840113Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9840273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9840659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9840841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9840847Z 2022-11-23T02:36:07.9840939Z Running tests... 2022-11-23T02:36:07.9841202Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9841466Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s) 2022-11-23T02:36:07.9841473Z 2022-11-23T02:36:07.9841762Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9841874Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9841880Z 2022-11-23T02:36:07.9841987Z OK (skipped=1) 2022-11-23T02:36:07.9841994Z 2022-11-23T02:36:07.9842120Z Generating XML reports... 2022-11-23T02:36:07.9842649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021822.xml 2022-11-23T02:36:07.9843028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9843469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9843657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9844111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9844320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9844327Z 2022-11-23T02:36:07.9844435Z Running tests... 2022-11-23T02:36:07.9844747Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9845046Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s) 2022-11-23T02:36:07.9845064Z 2022-11-23T02:36:07.9845363Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9845537Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9845544Z 2022-11-23T02:36:07.9845658Z OK (skipped=1) 2022-11-23T02:36:07.9845664Z 2022-11-23T02:36:07.9845789Z Generating XML reports... 2022-11-23T02:36:07.9846314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021826.xml 2022-11-23T02:36:07.9846687Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9847130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9847321Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9847777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9847985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9848045Z 2022-11-23T02:36:07.9848156Z Running tests... 2022-11-23T02:36:07.9848475Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9848779Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s) 2022-11-23T02:36:07.9848786Z 2022-11-23T02:36:07.9849094Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9849205Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9849212Z 2022-11-23T02:36:07.9849319Z OK (skipped=1) 2022-11-23T02:36:07.9849326Z 2022-11-23T02:36:07.9849455Z Generating XML reports... 2022-11-23T02:36:07.9849976Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021830.xml 2022-11-23T02:36:07.9850346Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9850928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9851129Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9851590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9851788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9851804Z 2022-11-23T02:36:07.9851904Z Running tests... 2022-11-23T02:36:07.9852216Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9852501Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s) 2022-11-23T02:36:07.9852508Z 2022-11-23T02:36:07.9852815Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9852927Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9852934Z 2022-11-23T02:36:07.9853040Z OK (skipped=1) 2022-11-23T02:36:07.9853046Z 2022-11-23T02:36:07.9853175Z Generating XML reports... 2022-11-23T02:36:07.9853701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021834.xml 2022-11-23T02:36:07.9854073Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9854513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9854703Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9855158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9855365Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9855372Z 2022-11-23T02:36:07.9855480Z Running tests... 2022-11-23T02:36:07.9855792Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9856159Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s) 2022-11-23T02:36:07.9856170Z 2022-11-23T02:36:07.9856490Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9856603Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9856610Z 2022-11-23T02:36:07.9856717Z OK (skipped=1) 2022-11-23T02:36:07.9856723Z 2022-11-23T02:36:07.9856849Z Generating XML reports... 2022-11-23T02:36:07.9857372Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021838.xml 2022-11-23T02:36:07.9857735Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9858177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9858371Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9858824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9859096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9859103Z 2022-11-23T02:36:07.9859212Z Running tests... 2022-11-23T02:36:07.9859540Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9859898Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49144 2022-11-23T02:36:07.9860104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49145 2022-11-23T02:36:07.9860356Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9860732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9860893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9861283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9861460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9861686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9862058Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9862220Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9862603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9862778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9863008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9863410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9863808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9864025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9864238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9864453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:36:07.9864848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:07.9865070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:36:07.9865466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:07.9865778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T02:36:07.9866001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T02:36:07.9866401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:36:07.9866796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T02:36:07.9866882Z ok (5.115s) 2022-11-23T02:36:07.9866888Z 2022-11-23T02:36:07.9867151Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9867249Z Ran 1 test in 5.116s 2022-11-23T02:36:07.9867255Z 2022-11-23T02:36:07.9867332Z OK 2022-11-23T02:36:07.9867338Z 2022-11-23T02:36:07.9867444Z Generating XML reports... 2022-11-23T02:36:07.9867889Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021843.xml 2022-11-23T02:36:07.9868256Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9868631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9868795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9869177Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9869353Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9869359Z 2022-11-23T02:36:07.9869450Z Running tests... 2022-11-23T02:36:07.9869720Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9870029Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49355 2022-11-23T02:36:07.9870242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49356 2022-11-23T02:36:07.9870487Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9870864Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9871028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9871410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9871588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9871812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9872184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9872353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9872737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9872913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9873137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9873537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9873931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9874145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9874357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9874687Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9874967Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9875054Z ok (5.636s) 2022-11-23T02:36:07.9875060Z 2022-11-23T02:36:07.9875322Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9875417Z Ran 1 test in 5.636s 2022-11-23T02:36:07.9875423Z 2022-11-23T02:36:07.9875500Z OK 2022-11-23T02:36:07.9875506Z 2022-11-23T02:36:07.9875613Z Generating XML reports... 2022-11-23T02:36:07.9876049Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021852.xml 2022-11-23T02:36:07.9876364Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9876738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9876950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9877338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9877519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9877525Z 2022-11-23T02:36:07.9877617Z Running tests... 2022-11-23T02:36:07.9877878Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9878203Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49569 2022-11-23T02:36:07.9878409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49570 2022-11-23T02:36:07.9878662Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9879039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9879206Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9879594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9879769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9879993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9880367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9880528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9880912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9881088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9881320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9881717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9882114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9882320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9882533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9882677Z skip: Skipped due to small world size. (4.909s) 2022-11-23T02:36:07.9882682Z 2022-11-23T02:36:07.9882946Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9883045Z Ran 1 test in 4.910s 2022-11-23T02:36:07.9883051Z 2022-11-23T02:36:07.9883145Z OK (skipped=1) 2022-11-23T02:36:07.9883151Z 2022-11-23T02:36:07.9883311Z Generating XML reports... 2022-11-23T02:36:07.9883759Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021902.xml 2022-11-23T02:36:07.9884072Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9884444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9884606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9884996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9885171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9885177Z 2022-11-23T02:36:07.9885269Z Running tests... 2022-11-23T02:36:07.9885532Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9885895Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49772 2022-11-23T02:36:07.9886100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49773 2022-11-23T02:36:07.9886353Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9886729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9886889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9887273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9887450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9887666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9888050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9888214Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9888597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9888777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9889003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9889403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9889798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9890014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9890228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9890313Z ok (5.327s) 2022-11-23T02:36:07.9890319Z 2022-11-23T02:36:07.9890701Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9890821Z Ran 1 test in 5.328s 2022-11-23T02:36:07.9890830Z 2022-11-23T02:36:07.9890946Z OK 2022-11-23T02:36:07.9890961Z 2022-11-23T02:36:07.9891175Z Generating XML reports... 2022-11-23T02:36:07.9891752Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021911.xml 2022-11-23T02:36:07.9892073Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9892445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9892605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9893075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9893254Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9893260Z 2022-11-23T02:36:07.9893351Z Running tests... 2022-11-23T02:36:07.9893610Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9893928Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49975 2022-11-23T02:36:07.9894134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49976 2022-11-23T02:36:07.9894388Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9894761Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9894976Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9895364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9895540Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9895763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9896132Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9896293Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9896677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9896852Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9897081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9897481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9897875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9898093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9898307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9898390Z ok (4.917s) 2022-11-23T02:36:07.9898396Z 2022-11-23T02:36:07.9898661Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9898756Z Ran 1 test in 4.917s 2022-11-23T02:36:07.9898762Z 2022-11-23T02:36:07.9898838Z OK 2022-11-23T02:36:07.9898844Z 2022-11-23T02:36:07.9898951Z Generating XML reports... 2022-11-23T02:36:07.9899392Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021920.xml 2022-11-23T02:36:07.9899709Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9900084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9900244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9900628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9900804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9900810Z 2022-11-23T02:36:07.9900901Z Running tests... 2022-11-23T02:36:07.9901164Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9901544Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50178 2022-11-23T02:36:07.9901754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50179 2022-11-23T02:36:07.9902006Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9902382Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9902543Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9902926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9903103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9903327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9903703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9903909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9904297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9904474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9904699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9905102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9905495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9905701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9905925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9906202Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9906474Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9906564Z ok (5.710s) 2022-11-23T02:36:07.9906570Z 2022-11-23T02:36:07.9906832Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9906927Z Ran 1 test in 5.711s 2022-11-23T02:36:07.9906933Z 2022-11-23T02:36:07.9907009Z OK 2022-11-23T02:36:07.9907014Z 2022-11-23T02:36:07.9907122Z Generating XML reports... 2022-11-23T02:36:07.9907565Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021929.xml 2022-11-23T02:36:07.9907879Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9908255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9908420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9908804Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9908979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9908984Z 2022-11-23T02:36:07.9909076Z Running tests... 2022-11-23T02:36:07.9909338Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9909653Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50388 2022-11-23T02:36:07.9909858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50389 2022-11-23T02:36:07.9910113Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9910557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9910713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9911102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9911278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9911501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9911874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9912034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9912419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9912646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9912869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9913272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9913664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9913878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9914093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9914368Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9914643Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9914730Z ok (5.709s) 2022-11-23T02:36:07.9914739Z 2022-11-23T02:36:07.9915006Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9915101Z Ran 1 test in 5.710s 2022-11-23T02:36:07.9915106Z 2022-11-23T02:36:07.9915182Z OK 2022-11-23T02:36:07.9915187Z 2022-11-23T02:36:07.9915294Z Generating XML reports... 2022-11-23T02:36:07.9915738Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021939.xml 2022-11-23T02:36:07.9916051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9916416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9916580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9916963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9917138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9917149Z 2022-11-23T02:36:07.9917241Z Running tests... 2022-11-23T02:36:07.9917506Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9917823Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50598 2022-11-23T02:36:07.9918033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50599 2022-11-23T02:36:07.9918286Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9918658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9918819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9919203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9919435Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9919665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9920041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9920200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9920585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9920761Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9920987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9921392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9921839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9922052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9922264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9922340Z ok (4.914s) 2022-11-23T02:36:07.9922346Z 2022-11-23T02:36:07.9922613Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9922709Z Ran 1 test in 4.914s 2022-11-23T02:36:07.9922714Z 2022-11-23T02:36:07.9922790Z OK 2022-11-23T02:36:07.9922796Z 2022-11-23T02:36:07.9922905Z Generating XML reports... 2022-11-23T02:36:07.9923347Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021949.xml 2022-11-23T02:36:07.9923661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9924040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9924200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9924589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9924765Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9924771Z 2022-11-23T02:36:07.9924863Z Running tests... 2022-11-23T02:36:07.9925142Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9925385Z test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:07.9925391Z 2022-11-23T02:36:07.9925651Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9925748Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9925757Z 2022-11-23T02:36:07.9925848Z OK (skipped=1) 2022-11-23T02:36:07.9925856Z 2022-11-23T02:36:07.9925967Z Generating XML reports... 2022-11-23T02:36:07.9926418Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021958.xml 2022-11-23T02:36:07.9926741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9927134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9927286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9927673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9927861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9927867Z 2022-11-23T02:36:07.9927975Z Running tests... 2022-11-23T02:36:07.9928256Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9929272Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.586s) 2022-11-23T02:36:07.9929280Z 2022-11-23T02:36:07.9929566Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9929673Z Ran 1 test in 0.586s 2022-11-23T02:36:07.9929679Z 2022-11-23T02:36:07.9929785Z OK (skipped=1) 2022-11-23T02:36:07.9929790Z 2022-11-23T02:36:07.9929902Z Generating XML reports... 2022-11-23T02:36:07.9930349Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022002.xml 2022-11-23T02:36:07.9930803Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9931257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9931423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9931823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9932014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9932020Z 2022-11-23T02:36:07.9932119Z Running tests... 2022-11-23T02:36:07.9932391Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9932651Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:07.9932657Z 2022-11-23T02:36:07.9932928Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9933029Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9933039Z 2022-11-23T02:36:07.9933142Z OK (skipped=1) 2022-11-23T02:36:07.9933147Z 2022-11-23T02:36:07.9933264Z Generating XML reports... 2022-11-23T02:36:07.9933715Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022007.xml 2022-11-23T02:36:07.9934033Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9934417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9934570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9934958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9935140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9935146Z 2022-11-23T02:36:07.9935245Z Running tests... 2022-11-23T02:36:07.9935526Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9935782Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:07.9935788Z 2022-11-23T02:36:07.9936054Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9936149Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9936155Z 2022-11-23T02:36:07.9936251Z OK (skipped=1) 2022-11-23T02:36:07.9936257Z 2022-11-23T02:36:07.9936375Z Generating XML reports... 2022-11-23T02:36:07.9936824Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022011.xml 2022-11-23T02:36:07.9937149Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9937526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9937696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9938144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9938329Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9938334Z 2022-11-23T02:36:07.9938435Z Running tests... 2022-11-23T02:36:07.9938713Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9938973Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s) 2022-11-23T02:36:07.9938979Z 2022-11-23T02:36:07.9939247Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9939347Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9939353Z 2022-11-23T02:36:07.9939451Z OK (skipped=1) 2022-11-23T02:36:07.9939457Z 2022-11-23T02:36:07.9939571Z Generating XML reports... 2022-11-23T02:36:07.9940004Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022015.xml 2022-11-23T02:36:07.9940372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9940756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9940923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9941314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9941495Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9941500Z 2022-11-23T02:36:07.9941600Z Running tests... 2022-11-23T02:36:07.9941872Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9942806Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.579s) 2022-11-23T02:36:07.9942816Z 2022-11-23T02:36:07.9943085Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9943191Z Ran 1 test in 0.579s 2022-11-23T02:36:07.9943197Z 2022-11-23T02:36:07.9943294Z OK (skipped=1) 2022-11-23T02:36:07.9943299Z 2022-11-23T02:36:07.9943415Z Generating XML reports... 2022-11-23T02:36:07.9943860Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022019.xml 2022-11-23T02:36:07.9944181Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9944562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9944739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9945132Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9945315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9945321Z 2022-11-23T02:36:07.9945415Z Running tests... 2022-11-23T02:36:07.9945680Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9946198Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:36:07.9946205Z 2022-11-23T02:36:07.9946473Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9946568Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9946574Z 2022-11-23T02:36:07.9946677Z OK (skipped=1) 2022-11-23T02:36:07.9946682Z 2022-11-23T02:36:07.9946801Z Generating XML reports... 2022-11-23T02:36:07.9947281Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022024.xml 2022-11-23T02:36:07.9947601Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9947975Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9948134Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9948517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9948695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9948701Z 2022-11-23T02:36:07.9948793Z Running tests... 2022-11-23T02:36:07.9949061Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9949587Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s) 2022-11-23T02:36:07.9949643Z 2022-11-23T02:36:07.9949912Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9950007Z Ran 1 test in 0.002s 2022-11-23T02:36:07.9950012Z 2022-11-23T02:36:07.9950103Z OK (skipped=1) 2022-11-23T02:36:07.9950109Z 2022-11-23T02:36:07.9950215Z Generating XML reports... 2022-11-23T02:36:07.9950656Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022029.xml 2022-11-23T02:36:07.9950969Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9951342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9951511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9951905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9952082Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9952088Z 2022-11-23T02:36:07.9952181Z Running tests... 2022-11-23T02:36:07.9952446Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9952757Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51329 2022-11-23T02:36:07.9952963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51330 2022-11-23T02:36:07.9953210Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9953587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9953754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9954152Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9954327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9954554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9954926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9955087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9955471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9955651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9955882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9956342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9956754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9956978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9957193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9957430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6zkvfe8 2022-11-23T02:36:07.9957687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6zkvfe8/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9957972Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9958228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoo4cgmm9 2022-11-23T02:36:07.9958539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoo4cgmm9/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9958828Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9959058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9959265Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9959368Z ok (7.623s) 2022-11-23T02:36:07.9959374Z 2022-11-23T02:36:07.9959657Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9959768Z Ran 1 test in 7.624s 2022-11-23T02:36:07.9959773Z 2022-11-23T02:36:07.9959862Z OK 2022-11-23T02:36:07.9959867Z 2022-11-23T02:36:07.9959988Z Generating XML reports... 2022-11-23T02:36:07.9960444Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022033.xml 2022-11-23T02:36:07.9960775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9961168Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9961341Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9961745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9961938Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9961944Z 2022-11-23T02:36:07.9962051Z Running tests... 2022-11-23T02:36:07.9962331Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9962657Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51549 2022-11-23T02:36:07.9962870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51550 2022-11-23T02:36:07.9963148Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9963533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9963707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9964099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9964277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9964513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9964877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9965053Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9965500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9965690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9965925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9966340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9966747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9966978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9967205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9967451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvfbyugla 2022-11-23T02:36:07.9967768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvfbyugla/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9968017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgxhyzvr 2022-11-23T02:36:07.9968279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgxhyzvr/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9968571Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9968844Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9969065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9969281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9969499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9969723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9969810Z ok (7.524s) 2022-11-23T02:36:07.9969816Z 2022-11-23T02:36:07.9970080Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9970167Z Ran 1 test in 7.525s 2022-11-23T02:36:07.9970182Z 2022-11-23T02:36:07.9970250Z OK 2022-11-23T02:36:07.9970255Z 2022-11-23T02:36:07.9970364Z Generating XML reports... 2022-11-23T02:36:07.9971173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022045.xml 2022-11-23T02:36:07.9971695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9972072Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9972235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9972622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9972801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9972807Z 2022-11-23T02:36:07.9972899Z Running tests... 2022-11-23T02:36:07.9973163Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9974091Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.582s) 2022-11-23T02:36:07.9974098Z 2022-11-23T02:36:07.9974359Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9974455Z Ran 1 test in 0.583s 2022-11-23T02:36:07.9974460Z 2022-11-23T02:36:07.9974556Z OK (skipped=1) 2022-11-23T02:36:07.9974561Z 2022-11-23T02:36:07.9974742Z Generating XML reports... 2022-11-23T02:36:07.9975193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022057.xml 2022-11-23T02:36:07.9975507Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9975881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9976041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9976426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9976602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9976608Z 2022-11-23T02:36:07.9976699Z Running tests... 2022-11-23T02:36:07.9976963Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9977994Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.589s) 2022-11-23T02:36:07.9978002Z 2022-11-23T02:36:07.9978265Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9978360Z Ran 1 test in 0.589s 2022-11-23T02:36:07.9978366Z 2022-11-23T02:36:07.9978459Z OK (skipped=1) 2022-11-23T02:36:07.9978464Z 2022-11-23T02:36:07.9978572Z Generating XML reports... 2022-11-23T02:36:07.9979003Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022101.xml 2022-11-23T02:36:07.9979319Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9979700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9979860Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9980245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9980421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9980427Z 2022-11-23T02:36:07.9980519Z Running tests... 2022-11-23T02:36:07.9980785Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9981259Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.003s) 2022-11-23T02:36:07.9981266Z 2022-11-23T02:36:07.9981526Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9981623Z Ran 1 test in 0.004s 2022-11-23T02:36:07.9981632Z 2022-11-23T02:36:07.9981724Z OK (skipped=1) 2022-11-23T02:36:07.9981729Z 2022-11-23T02:36:07.9981836Z Generating XML reports... 2022-11-23T02:36:07.9982280Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022106.xml 2022-11-23T02:36:07.9982593Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9982968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9983128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9983512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9983686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9983692Z 2022-11-23T02:36:07.9983788Z Running tests... 2022-11-23T02:36:07.9984103Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9984455Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51967 2022-11-23T02:36:07.9984662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51968 2022-11-23T02:36:07.9984915Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9985282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9985443Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9985824Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9985998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9986273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9986649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9986810Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9987198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9987373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9987600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9988001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9988398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9988613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9988823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9989059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyuksl_rg 2022-11-23T02:36:07.9989307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyuksl_rg/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9989584Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9989818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxt2_wy1e 2022-11-23T02:36:07.9990065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxt2_wy1e/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9990342Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9990429Z ok (5.312s) 2022-11-23T02:36:07.9990438Z 2022-11-23T02:36:07.9990700Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9990788Z Ran 1 test in 5.313s 2022-11-23T02:36:07.9990793Z 2022-11-23T02:36:07.9990870Z OK 2022-11-23T02:36:07.9990875Z 2022-11-23T02:36:07.9990983Z Generating XML reports... 2022-11-23T02:36:07.9991425Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022111.xml 2022-11-23T02:36:07.9991739Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:07.9992115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9992277Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9992664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9992892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9992898Z 2022-11-23T02:36:07.9992991Z Running tests... 2022-11-23T02:36:07.9993261Z ---------------------------------------------------------------------- 2022-11-23T02:36:07.9993568Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52177 2022-11-23T02:36:07.9993773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52178 2022-11-23T02:36:07.9994037Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:07.9994409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9994570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9994957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9995181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9995404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:07.9995777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:07.9995936Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:07.9996319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:07.9996485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:07.9996710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:07.9997109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9997512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:07.9997724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:07.9997940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:07.9998174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvpdeje6 2022-11-23T02:36:07.9998425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvpdeje6/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9998666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwwxd2kz 2022-11-23T02:36:07.9998914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwwxd2kz/_remote_module_non_scriptable.py 2022-11-23T02:36:07.9999188Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9999466Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:07.9999684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9999903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:07.9999987Z ok (7.514s) 2022-11-23T02:36:07.9999993Z 2022-11-23T02:36:08.0000256Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0000355Z Ran 1 test in 7.514s 2022-11-23T02:36:08.0000361Z 2022-11-23T02:36:08.0000438Z OK 2022-11-23T02:36:08.0000444Z 2022-11-23T02:36:08.0000552Z Generating XML reports... 2022-11-23T02:36:08.0000993Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022120.xml 2022-11-23T02:36:08.0001304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0001721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0001890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0002281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0002457Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0002463Z 2022-11-23T02:36:08.0002556Z Running tests... 2022-11-23T02:36:08.0002820Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0003298Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.005s) 2022-11-23T02:36:08.0003305Z 2022-11-23T02:36:08.0003573Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0003667Z Ran 1 test in 0.005s 2022-11-23T02:36:08.0003719Z 2022-11-23T02:36:08.0003816Z OK (skipped=1) 2022-11-23T02:36:08.0003822Z 2022-11-23T02:36:08.0003930Z Generating XML reports... 2022-11-23T02:36:08.0004377Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022132.xml 2022-11-23T02:36:08.0004693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0005071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0005240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0005623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0005797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0005803Z 2022-11-23T02:36:08.0005898Z Running tests... 2022-11-23T02:36:08.0006177Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0006638Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.004s) 2022-11-23T02:36:08.0006644Z 2022-11-23T02:36:08.0006910Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0007018Z Ran 1 test in 0.004s 2022-11-23T02:36:08.0007024Z 2022-11-23T02:36:08.0007126Z OK (skipped=1) 2022-11-23T02:36:08.0007132Z 2022-11-23T02:36:08.0007255Z Generating XML reports... 2022-11-23T02:36:08.0007687Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022136.xml 2022-11-23T02:36:08.0008019Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0008406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0008586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0008980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0009169Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0009175Z 2022-11-23T02:36:08.0009285Z Running tests... 2022-11-23T02:36:08.0009561Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0009900Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s) 2022-11-23T02:36:08.0009906Z 2022-11-23T02:36:08.0010180Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0010286Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0010292Z 2022-11-23T02:36:08.0010395Z OK (skipped=1) 2022-11-23T02:36:08.0010400Z 2022-11-23T02:36:08.0010582Z Generating XML reports... 2022-11-23T02:36:08.0011091Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022140.xml 2022-11-23T02:36:08.0011430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0011821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0011993Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0012393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0012579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0012585Z 2022-11-23T02:36:08.0012692Z Running tests... 2022-11-23T02:36:08.0012971Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0013402Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.005s) 2022-11-23T02:36:08.0013461Z 2022-11-23T02:36:08.0013742Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0013829Z Ran 1 test in 0.005s 2022-11-23T02:36:08.0013834Z 2022-11-23T02:36:08.0013940Z OK (skipped=1) 2022-11-23T02:36:08.0013945Z 2022-11-23T02:36:08.0014064Z Generating XML reports... 2022-11-23T02:36:08.0014521Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022144.xml 2022-11-23T02:36:08.0014848Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0015233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0015408Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0015810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0016003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0016009Z 2022-11-23T02:36:08.0016120Z Running tests... 2022-11-23T02:36:08.0016400Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0016867Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s) 2022-11-23T02:36:08.0016873Z 2022-11-23T02:36:08.0017148Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0017254Z Ran 1 test in 0.003s 2022-11-23T02:36:08.0017260Z 2022-11-23T02:36:08.0017365Z OK (skipped=1) 2022-11-23T02:36:08.0017370Z 2022-11-23T02:36:08.0017494Z Generating XML reports... 2022-11-23T02:36:08.0017944Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022148.xml 2022-11-23T02:36:08.0018269Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0018655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0018831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0019230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0019418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0019424Z 2022-11-23T02:36:08.0019533Z Running tests... 2022-11-23T02:36:08.0019786Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0020785Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.583s) 2022-11-23T02:36:08.0020825Z 2022-11-23T02:36:08.0021111Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0021198Z Ran 1 test in 0.583s 2022-11-23T02:36:08.0021204Z 2022-11-23T02:36:08.0021312Z OK (skipped=1) 2022-11-23T02:36:08.0021317Z 2022-11-23T02:36:08.0021437Z Generating XML reports... 2022-11-23T02:36:08.0021892Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022152.xml 2022-11-23T02:36:08.0022224Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0022616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0022789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0023190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0023430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0023435Z 2022-11-23T02:36:08.0023544Z Running tests... 2022-11-23T02:36:08.0023827Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0024773Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.578s) 2022-11-23T02:36:08.0024780Z 2022-11-23T02:36:08.0025056Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0025166Z Ran 1 test in 0.578s 2022-11-23T02:36:08.0025172Z 2022-11-23T02:36:08.0025284Z OK (skipped=1) 2022-11-23T02:36:08.0025289Z 2022-11-23T02:36:08.0025416Z Generating XML reports... 2022-11-23T02:36:08.0025872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022157.xml 2022-11-23T02:36:08.0026206Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0026596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0026774Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0027175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0027367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0027373Z 2022-11-23T02:36:08.0027482Z Running tests... 2022-11-23T02:36:08.0027763Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0028116Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52859 2022-11-23T02:36:08.0028313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52860 2022-11-23T02:36:08.0028581Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0028969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0029153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0029552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0029740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0029980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0030419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0030600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0031001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0031192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0031433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0031848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0032260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0032482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0032758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:36:08.0032985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0033212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:36:08.0033629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0034031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0034282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2ezx1j0 2022-11-23T02:36:08.0034531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2ezx1j0/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0034782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyx91vhe 2022-11-23T02:36:08.0035043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyx91vhe/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0035309Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0035580Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0035802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0036036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0036250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0036475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0036576Z ok (8.016s) 2022-11-23T02:36:08.0036582Z 2022-11-23T02:36:08.0036848Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0036953Z Ran 1 test in 8.016s 2022-11-23T02:36:08.0036959Z 2022-11-23T02:36:08.0037048Z OK 2022-11-23T02:36:08.0037054Z 2022-11-23T02:36:08.0037160Z Generating XML reports... 2022-11-23T02:36:08.0037605Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022202.xml 2022-11-23T02:36:08.0037928Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0038302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0038473Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0038874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0039053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0039062Z 2022-11-23T02:36:08.0039218Z Running tests... 2022-11-23T02:36:08.0039488Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0039815Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53081 2022-11-23T02:36:08.0040036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53082 2022-11-23T02:36:08.0040280Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0040652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0040821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0041221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0041471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0041710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0042087Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0042263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0042652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0042843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0043074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0043489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0043896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0044133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0044395Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:36:08.0044614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0044889Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:36:08.0045126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbdqky5g 2022-11-23T02:36:08.0045389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbdqky5g/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0045674Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0045925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpir42y0aw 2022-11-23T02:36:08.0046176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpir42y0aw/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0046460Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0046669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0046891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0047123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0047341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0047623Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:36:08.0047884Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:36:08.0048222Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:36:08.0048490Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-11-23T02:36:08.0048716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0048939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0049168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0049387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0049659Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:36:08.0049922Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-11-23T02:36:08.0050237Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:36:08.0050498Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-11-23T02:36:08.0050888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0051112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0051334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0051559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0051662Z ok (8.427s) 2022-11-23T02:36:08.0051668Z 2022-11-23T02:36:08.0051948Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0052041Z Ran 1 test in 8.427s 2022-11-23T02:36:08.0052054Z 2022-11-23T02:36:08.0052125Z OK 2022-11-23T02:36:08.0052131Z 2022-11-23T02:36:08.0052238Z Generating XML reports... 2022-11-23T02:36:08.0052683Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022214.xml 2022-11-23T02:36:08.0052996Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0053375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0053548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0053931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0054106Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0054112Z 2022-11-23T02:36:08.0054203Z Running tests... 2022-11-23T02:36:08.0054472Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0055400Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.598s) 2022-11-23T02:36:08.0055407Z 2022-11-23T02:36:08.0055668Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0055763Z Ran 1 test in 0.599s 2022-11-23T02:36:08.0055769Z 2022-11-23T02:36:08.0055860Z OK (skipped=1) 2022-11-23T02:36:08.0055866Z 2022-11-23T02:36:08.0055973Z Generating XML reports... 2022-11-23T02:36:08.0056415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022227.xml 2022-11-23T02:36:08.0056800Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0057181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0057342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0057728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0057904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0057910Z 2022-11-23T02:36:08.0058001Z Running tests... 2022-11-23T02:36:08.0058264Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0058582Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53367 2022-11-23T02:36:08.0058779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53368 2022-11-23T02:36:08.0059090Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0059469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0059630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0060014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0060189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0060412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0060783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0060943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0061334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0061508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0061732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0062132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0062528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0062744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0063278Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:36:08.0063495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0064021Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:36:08.0064256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xfaxaau 2022-11-23T02:36:08.0064505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xfaxaau/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0064738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2izxd37e 2022-11-23T02:36:08.0065037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2izxd37e/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0065319Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0065591Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0065809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0066025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0066283Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:36:08.0066538Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations. 2022-11-23T02:36:08.0066810Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:36:08.0067122Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:36:08.0067473Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:36:08.0067753Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-11-23T02:36:08.0068062Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0 2022-11-23T02:36:08.0068366Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-11-23T02:36:08.0068583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0068799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0068891Z ok (7.827s) 2022-11-23T02:36:08.0068897Z 2022-11-23T02:36:08.0069172Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0069271Z Ran 1 test in 7.828s 2022-11-23T02:36:08.0069276Z 2022-11-23T02:36:08.0069352Z OK 2022-11-23T02:36:08.0069357Z 2022-11-23T02:36:08.0069466Z Generating XML reports... 2022-11-23T02:36:08.0069914Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022231.xml 2022-11-23T02:36:08.0070229Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0070604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0070766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0071151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0071333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0071338Z 2022-11-23T02:36:08.0071432Z Running tests... 2022-11-23T02:36:08.0071696Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0072076Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0072082Z 2022-11-23T02:36:08.0072345Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0072441Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0072446Z 2022-11-23T02:36:08.0072539Z OK (skipped=1) 2022-11-23T02:36:08.0072544Z 2022-11-23T02:36:08.0072651Z Generating XML reports... 2022-11-23T02:36:08.0073086Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022243.xml 2022-11-23T02:36:08.0073487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0073870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0074030Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0074418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0074594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0074600Z 2022-11-23T02:36:08.0074691Z Running tests... 2022-11-23T02:36:08.0074956Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0075331Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0075381Z 2022-11-23T02:36:08.0075651Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0075750Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0075755Z 2022-11-23T02:36:08.0075846Z OK (skipped=1) 2022-11-23T02:36:08.0075852Z 2022-11-23T02:36:08.0075960Z Generating XML reports... 2022-11-23T02:36:08.0076403Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022247.xml 2022-11-23T02:36:08.0076717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0077092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0077254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0077638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0077820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0077826Z 2022-11-23T02:36:08.0077917Z Running tests... 2022-11-23T02:36:08.0078183Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0078619Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0078626Z 2022-11-23T02:36:08.0078888Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0078986Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0078992Z 2022-11-23T02:36:08.0079084Z OK (skipped=1) 2022-11-23T02:36:08.0079090Z 2022-11-23T02:36:08.0079189Z Generating XML reports... 2022-11-23T02:36:08.0079629Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022252.xml 2022-11-23T02:36:08.0079948Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0080322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0080487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0080874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0081049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0081055Z 2022-11-23T02:36:08.0081150Z Running tests... 2022-11-23T02:36:08.0081412Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0081898Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0081908Z 2022-11-23T02:36:08.0082174Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0082272Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0082278Z 2022-11-23T02:36:08.0082371Z OK (skipped=1) 2022-11-23T02:36:08.0082376Z 2022-11-23T02:36:08.0082483Z Generating XML reports... 2022-11-23T02:36:08.0082925Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022256.xml 2022-11-23T02:36:08.0083238Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0083611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0083775Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0084160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0084483Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0084489Z 2022-11-23T02:36:08.0084583Z Running tests... 2022-11-23T02:36:08.0084852Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0085281Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0085288Z 2022-11-23T02:36:08.0085550Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0085644Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0085650Z 2022-11-23T02:36:08.0085741Z OK (skipped=1) 2022-11-23T02:36:08.0085747Z 2022-11-23T02:36:08.0085845Z Generating XML reports... 2022-11-23T02:36:08.0086286Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022300.xml 2022-11-23T02:36:08.0086606Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0086980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0087142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0087526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0087700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0087706Z 2022-11-23T02:36:08.0087798Z Running tests... 2022-11-23T02:36:08.0088062Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0088493Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0088503Z 2022-11-23T02:36:08.0088768Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0088869Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0088874Z 2022-11-23T02:36:08.0088965Z OK (skipped=1) 2022-11-23T02:36:08.0088970Z 2022-11-23T02:36:08.0089077Z Generating XML reports... 2022-11-23T02:36:08.0089515Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022304.xml 2022-11-23T02:36:08.0089828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0090201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0090361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0090853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0091040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0091046Z 2022-11-23T02:36:08.0091138Z Running tests... 2022-11-23T02:36:08.0091406Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0091838Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0091844Z 2022-11-23T02:36:08.0092106Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0092201Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0092206Z 2022-11-23T02:36:08.0092288Z OK (skipped=1) 2022-11-23T02:36:08.0092302Z 2022-11-23T02:36:08.0092401Z Generating XML reports... 2022-11-23T02:36:08.0092903Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022308.xml 2022-11-23T02:36:08.0093218Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0093592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0093755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0094139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0094314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0094320Z 2022-11-23T02:36:08.0094411Z Running tests... 2022-11-23T02:36:08.0094675Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0095106Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0095114Z 2022-11-23T02:36:08.0095377Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0095472Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0095478Z 2022-11-23T02:36:08.0095569Z OK (skipped=1) 2022-11-23T02:36:08.0095574Z 2022-11-23T02:36:08.0095682Z Generating XML reports... 2022-11-23T02:36:08.0096121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022312.xml 2022-11-23T02:36:08.0096433Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0096805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0096968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0097360Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0097534Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0097540Z 2022-11-23T02:36:08.0097631Z Running tests... 2022-11-23T02:36:08.0097893Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0098318Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0098324Z 2022-11-23T02:36:08.0098586Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0098682Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0098687Z 2022-11-23T02:36:08.0098769Z OK (skipped=1) 2022-11-23T02:36:08.0098783Z 2022-11-23T02:36:08.0098882Z Generating XML reports... 2022-11-23T02:36:08.0099376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022316.xml 2022-11-23T02:36:08.0099696Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0100070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0100235Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0100621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0100797Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0100803Z 2022-11-23T02:36:08.0100900Z Running tests... 2022-11-23T02:36:08.0101182Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0101606Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0101657Z 2022-11-23T02:36:08.0101926Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0102032Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0102037Z 2022-11-23T02:36:08.0102137Z OK (skipped=1) 2022-11-23T02:36:08.0102143Z 2022-11-23T02:36:08.0102263Z Generating XML reports... 2022-11-23T02:36:08.0102717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022320.xml 2022-11-23T02:36:08.0103039Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0103415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0103576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0103964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0104138Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0104144Z 2022-11-23T02:36:08.0104239Z Running tests... 2022-11-23T02:36:08.0104503Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0104879Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0104886Z 2022-11-23T02:36:08.0105149Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0105237Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0105253Z 2022-11-23T02:36:08.0105336Z OK (skipped=1) 2022-11-23T02:36:08.0105341Z 2022-11-23T02:36:08.0105451Z Generating XML reports... 2022-11-23T02:36:08.0105898Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022324.xml 2022-11-23T02:36:08.0106213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0106587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0106749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0107135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0107311Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0107317Z 2022-11-23T02:36:08.0107409Z Running tests... 2022-11-23T02:36:08.0107673Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0108096Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s) 2022-11-23T02:36:08.0108106Z 2022-11-23T02:36:08.0108373Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0108471Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0108477Z 2022-11-23T02:36:08.0108568Z OK (skipped=1) 2022-11-23T02:36:08.0108574Z 2022-11-23T02:36:08.0108681Z Generating XML reports... 2022-11-23T02:36:08.0109121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022328.xml 2022-11-23T02:36:08.0109436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0109810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0109971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0110428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0110604Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0110610Z 2022-11-23T02:36:08.0110703Z Running tests... 2022-11-23T02:36:08.0110958Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0111395Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:36:08.0111409Z 2022-11-23T02:36:08.0111661Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0111758Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0111763Z 2022-11-23T02:36:08.0111857Z OK (skipped=1) 2022-11-23T02:36:08.0111862Z 2022-11-23T02:36:08.0111968Z Generating XML reports... 2022-11-23T02:36:08.0112409Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022332.xml 2022-11-23T02:36:08.0112728Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0113101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0113264Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0113649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0113825Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0113831Z 2022-11-23T02:36:08.0113925Z Running tests... 2022-11-23T02:36:08.0114187Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0114489Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54445 2022-11-23T02:36:08.0114700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54446 2022-11-23T02:36:08.0114959Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0115331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0115494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0115879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0116057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0116280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0116654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0116866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0117248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0117423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0117647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0118051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0118449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0118668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0118882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0119167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5imasbpx 2022-11-23T02:36:08.0119417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5imasbpx/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0119650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr430pzr 2022-11-23T02:36:08.0119897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr430pzr/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0120177Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0120450Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0120537Z ok (7.425s) 2022-11-23T02:36:08.0120543Z 2022-11-23T02:36:08.0120807Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0120903Z Ran 1 test in 7.425s 2022-11-23T02:36:08.0120909Z 2022-11-23T02:36:08.0120985Z OK 2022-11-23T02:36:08.0120991Z 2022-11-23T02:36:08.0121101Z Generating XML reports... 2022-11-23T02:36:08.0121545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022336.xml 2022-11-23T02:36:08.0121860Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0122235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0122388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0122770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0122946Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0122951Z 2022-11-23T02:36:08.0123045Z Running tests... 2022-11-23T02:36:08.0123307Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0123632Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54658 2022-11-23T02:36:08.0123841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54659 2022-11-23T02:36:08.0124095Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0124471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0124635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0125018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0125193Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0125418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0125845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0126007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0126396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0126572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0126800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0127199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0127594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0127809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0128071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0128306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn596ynb0 2022-11-23T02:36:08.0128544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn596ynb0/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0128774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6cfztr28 2022-11-23T02:36:08.0129018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6cfztr28/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0129299Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0129571Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0129791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0130008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0130415Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:36:08.0130643Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:36:08.0131371Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T02:36:08.0131638Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T02:36:08.0131725Z ok (7.426s) 2022-11-23T02:36:08.0131731Z 2022-11-23T02:36:08.0132002Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0132098Z Ran 1 test in 7.427s 2022-11-23T02:36:08.0132104Z 2022-11-23T02:36:08.0132179Z OK 2022-11-23T02:36:08.0132184Z 2022-11-23T02:36:08.0132292Z Generating XML reports... 2022-11-23T02:36:08.0132735Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022348.xml 2022-11-23T02:36:08.0133058Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0133433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0133598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0133992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0134160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0134175Z 2022-11-23T02:36:08.0134258Z Running tests... 2022-11-23T02:36:08.0134527Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0134789Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.007s) 2022-11-23T02:36:08.0134795Z 2022-11-23T02:36:08.0135058Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0135226Z Ran 1 test in 0.007s 2022-11-23T02:36:08.0135234Z 2022-11-23T02:36:08.0135325Z OK (skipped=1) 2022-11-23T02:36:08.0135331Z 2022-11-23T02:36:08.0135443Z Generating XML reports... 2022-11-23T02:36:08.0135891Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022359.xml 2022-11-23T02:36:08.0136205Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0136578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0136738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0137125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0137302Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0137360Z 2022-11-23T02:36:08.0137455Z Running tests... 2022-11-23T02:36:08.0137730Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0138039Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54944 2022-11-23T02:36:08.0138245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54945 2022-11-23T02:36:08.0138498Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0138871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0139032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0139416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0139598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0139817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0140195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0140355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0140741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0140918Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0141144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0141545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0141946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0142166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0142380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0142618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2ktq1r_ 2022-11-23T02:36:08.0142867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2ktq1r_/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0143100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2vgopku 2022-11-23T02:36:08.0143348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2vgopku/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0143627Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0143903Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0144176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0144397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0144483Z ok (8.368s) 2022-11-23T02:36:08.0144489Z 2022-11-23T02:36:08.0144760Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0144855Z Ran 1 test in 8.368s 2022-11-23T02:36:08.0144861Z 2022-11-23T02:36:08.0144929Z OK 2022-11-23T02:36:08.0144945Z 2022-11-23T02:36:08.0145043Z Generating XML reports... 2022-11-23T02:36:08.0145492Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022403.xml 2022-11-23T02:36:08.0145807Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0146182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0146397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0146786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0146962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0146968Z 2022-11-23T02:36:08.0147059Z Running tests... 2022-11-23T02:36:08.0147324Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0147793Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:36:08.0147799Z 2022-11-23T02:36:08.0148062Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0148159Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0148165Z 2022-11-23T02:36:08.0148255Z OK (skipped=1) 2022-11-23T02:36:08.0148261Z 2022-11-23T02:36:08.0148373Z Generating XML reports... 2022-11-23T02:36:08.0148821Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022416.xml 2022-11-23T02:36:08.0149138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0149513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0149673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0150058Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0150233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0150239Z 2022-11-23T02:36:08.0150333Z Running tests... 2022-11-23T02:36:08.0150597Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0151055Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:36:08.0151074Z 2022-11-23T02:36:08.0151327Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0151423Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0151429Z 2022-11-23T02:36:08.0151520Z OK (skipped=1) 2022-11-23T02:36:08.0151525Z 2022-11-23T02:36:08.0151632Z Generating XML reports... 2022-11-23T02:36:08.0152075Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022420.xml 2022-11-23T02:36:08.0152391Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0152766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0152926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0153364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0153541Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0153547Z 2022-11-23T02:36:08.0153638Z Running tests... 2022-11-23T02:36:08.0153905Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0154408Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s) 2022-11-23T02:36:08.0154414Z 2022-11-23T02:36:08.0154676Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0154773Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0154779Z 2022-11-23T02:36:08.0154873Z OK (skipped=1) 2022-11-23T02:36:08.0154878Z 2022-11-23T02:36:08.0154987Z Generating XML reports... 2022-11-23T02:36:08.0155430Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022424.xml 2022-11-23T02:36:08.0155802Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0156175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0156337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0156724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0156901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0156907Z 2022-11-23T02:36:08.0156990Z Running tests... 2022-11-23T02:36:08.0157256Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0157738Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s) 2022-11-23T02:36:08.0157747Z 2022-11-23T02:36:08.0158015Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0158113Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0158119Z 2022-11-23T02:36:08.0158210Z OK (skipped=1) 2022-11-23T02:36:08.0158216Z 2022-11-23T02:36:08.0158323Z Generating XML reports... 2022-11-23T02:36:08.0158764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022428.xml 2022-11-23T02:36:08.0159083Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0159459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0159618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0160004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0160185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0160191Z 2022-11-23T02:36:08.0160283Z Running tests... 2022-11-23T02:36:08.0160548Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0160971Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.003s) 2022-11-23T02:36:08.0160977Z 2022-11-23T02:36:08.0161241Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0161336Z Ran 1 test in 0.003s 2022-11-23T02:36:08.0161342Z 2022-11-23T02:36:08.0161433Z OK (skipped=1) 2022-11-23T02:36:08.0161439Z 2022-11-23T02:36:08.0161547Z Generating XML reports... 2022-11-23T02:36:08.0161987Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022432.xml 2022-11-23T02:36:08.0162302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0162729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0162882Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0163273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0163452Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0163458Z 2022-11-23T02:36:08.0163548Z Running tests... 2022-11-23T02:36:08.0163811Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0164117Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55494 2022-11-23T02:36:08.0164322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55495 2022-11-23T02:36:08.0164631Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0165010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0165169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0165555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0165731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0165956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0166329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0166492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0166881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0167062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0167290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0167695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0168093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0168309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0168527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0168766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapvb49_1 2022-11-23T02:36:08.0169011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapvb49_1/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0169290Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0169525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_mx80ds 2022-11-23T02:36:08.0169768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_mx80ds/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0170044Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0170997Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:36:08.0171782Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:36:08.0172003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0172218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0172295Z ok (7.544s) 2022-11-23T02:36:08.0172309Z 2022-11-23T02:36:08.0172577Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0172722Z Ran 1 test in 7.545s 2022-11-23T02:36:08.0172728Z 2022-11-23T02:36:08.0172804Z OK 2022-11-23T02:36:08.0172810Z 2022-11-23T02:36:08.0172916Z Generating XML reports... 2022-11-23T02:36:08.0173363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022436.xml 2022-11-23T02:36:08.0173678Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0174056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0174217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0174601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0174775Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0174784Z 2022-11-23T02:36:08.0174878Z Running tests... 2022-11-23T02:36:08.0175144Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0176094Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.589s) 2022-11-23T02:36:08.0176101Z 2022-11-23T02:36:08.0176360Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0176458Z Ran 1 test in 0.590s 2022-11-23T02:36:08.0176464Z 2022-11-23T02:36:08.0176554Z OK (skipped=1) 2022-11-23T02:36:08.0176560Z 2022-11-23T02:36:08.0176667Z Generating XML reports... 2022-11-23T02:36:08.0177111Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022448.xml 2022-11-23T02:36:08.0177431Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0177806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0177967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0178354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0178531Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0178537Z 2022-11-23T02:36:08.0178628Z Running tests... 2022-11-23T02:36:08.0178885Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0179345Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:36:08.0179364Z 2022-11-23T02:36:08.0179687Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0179785Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0179791Z 2022-11-23T02:36:08.0179882Z OK (skipped=1) 2022-11-23T02:36:08.0179888Z 2022-11-23T02:36:08.0179995Z Generating XML reports... 2022-11-23T02:36:08.0180444Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022453.xml 2022-11-23T02:36:08.0180762Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0181137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0181301Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0181691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0181866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0181919Z 2022-11-23T02:36:08.0182012Z Running tests... 2022-11-23T02:36:08.0182279Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0182732Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s) 2022-11-23T02:36:08.0182738Z 2022-11-23T02:36:08.0183003Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0183099Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0183105Z 2022-11-23T02:36:08.0183196Z OK (skipped=1) 2022-11-23T02:36:08.0183202Z 2022-11-23T02:36:08.0183309Z Generating XML reports... 2022-11-23T02:36:08.0183750Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022457.xml 2022-11-23T02:36:08.0184065Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0184444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0184605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0184984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0185165Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0185179Z 2022-11-23T02:36:08.0185262Z Running tests... 2022-11-23T02:36:08.0185527Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0185838Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55912 2022-11-23T02:36:08.0186047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55913 2022-11-23T02:36:08.0186304Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0186681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0186842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0187227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0187401Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0187629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0188004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0188165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0188600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0188781Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0189009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0189413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0189813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0190027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0190242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0190478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_xdoe8y 2022-11-23T02:36:08.0190728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_xdoe8y/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0190996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ti6y4wi 2022-11-23T02:36:08.0191244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ti6y4wi/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0191523Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0191798Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0191883Z ok (5.325s) 2022-11-23T02:36:08.0191889Z 2022-11-23T02:36:08.0192152Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0192246Z Ran 1 test in 5.326s 2022-11-23T02:36:08.0192252Z 2022-11-23T02:36:08.0192329Z OK 2022-11-23T02:36:08.0192334Z 2022-11-23T02:36:08.0192442Z Generating XML reports... 2022-11-23T02:36:08.0192886Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022501.xml 2022-11-23T02:36:08.0193208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0193585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0193745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0194133Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0194309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0194315Z 2022-11-23T02:36:08.0194408Z Running tests... 2022-11-23T02:36:08.0194672Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0195613Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.641s) 2022-11-23T02:36:08.0195626Z 2022-11-23T02:36:08.0195887Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0195983Z Ran 1 test in 0.641s 2022-11-23T02:36:08.0195989Z 2022-11-23T02:36:08.0196079Z OK (skipped=1) 2022-11-23T02:36:08.0196084Z 2022-11-23T02:36:08.0196192Z Generating XML reports... 2022-11-23T02:36:08.0196635Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022510.xml 2022-11-23T02:36:08.0196951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0197323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0197476Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0197922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0198101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0198106Z 2022-11-23T02:36:08.0198197Z Running tests... 2022-11-23T02:36:08.0198463Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0198929Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.003s) 2022-11-23T02:36:08.0198935Z 2022-11-23T02:36:08.0199198Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0199292Z Ran 1 test in 0.003s 2022-11-23T02:36:08.0199298Z 2022-11-23T02:36:08.0199389Z OK (skipped=1) 2022-11-23T02:36:08.0199395Z 2022-11-23T02:36:08.0199503Z Generating XML reports... 2022-11-23T02:36:08.0199948Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022515.xml 2022-11-23T02:36:08.0200315Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0200690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0200849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0201235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0201411Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0201417Z 2022-11-23T02:36:08.0201509Z Running tests... 2022-11-23T02:36:08.0201772Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0202717Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.612s) 2022-11-23T02:36:08.0202727Z 2022-11-23T02:36:08.0202991Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0203087Z Ran 1 test in 0.612s 2022-11-23T02:36:08.0203092Z 2022-11-23T02:36:08.0203183Z OK (skipped=1) 2022-11-23T02:36:08.0203188Z 2022-11-23T02:36:08.0203296Z Generating XML reports... 2022-11-23T02:36:08.0203743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022520.xml 2022-11-23T02:36:08.0204058Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0204433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0204591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0204976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0205153Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0205159Z 2022-11-23T02:36:08.0205249Z Running tests... 2022-11-23T02:36:08.0205513Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0205828Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56320 2022-11-23T02:36:08.0206035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56321 2022-11-23T02:36:08.0206291Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0206720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0206884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0207272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0207446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0207670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0208044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0208209Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0208594Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0208770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0209046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0209453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0209848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0210063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0210278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0210512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6kp5_kb 2022-11-23T02:36:08.0210805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6kp5_kb/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0211042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgd778qjn 2022-11-23T02:36:08.0211289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgd778qjn/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0211571Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0211846Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0212187Z STAGE:2022-11-23 02:25:28 56320:56320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0212518Z STAGE:2022-11-23 02:25:28 56321:56321 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0213161Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:08.0213256Z warnings.warn( 2022-11-23T02:36:08.0213474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0214118Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:36:08.0214211Z warnings.warn( 2022-11-23T02:36:08.0214430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:36:08.0214772Z STAGE:2022-11-23 02:25:28 56320:56320 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0215112Z STAGE:2022-11-23 02:25:28 56321:56321 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0215464Z STAGE:2022-11-23 02:25:28 56321:56321 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0215818Z STAGE:2022-11-23 02:25:28 56320:56320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0216127Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:36:08.0216353Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 5 (function operator()) 2022-11-23T02:36:08.0216694Z STAGE:2022-11-23 02:25:28 56320:56320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0217032Z STAGE:2022-11-23 02:25:28 56320:56320 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0217384Z STAGE:2022-11-23 02:25:28 56320:56320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0217468Z ok (5.738s) 2022-11-23T02:36:08.0217474Z 2022-11-23T02:36:08.0217731Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0217828Z Ran 1 test in 5.738s 2022-11-23T02:36:08.0217834Z 2022-11-23T02:36:08.0217909Z OK 2022-11-23T02:36:08.0217915Z 2022-11-23T02:36:08.0218023Z Generating XML reports... 2022-11-23T02:36:08.0218528Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022525.xml 2022-11-23T02:36:08.0218844Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0219222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0219383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0219771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0219947Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0219953Z 2022-11-23T02:36:08.0220045Z Running tests... 2022-11-23T02:36:08.0220312Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0220620Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56544 2022-11-23T02:36:08.0220828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56545 2022-11-23T02:36:08.0221083Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0221459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0221621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0222007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0222182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0222406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0222784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0222948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0223325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0223500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0223728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0224127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0224523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0224741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0225010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0225252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo82h6pdx 2022-11-23T02:36:08.0225499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo82h6pdx/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0225730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jtwjt4z 2022-11-23T02:36:08.0225976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jtwjt4z/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0226258Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0226533Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0226619Z ok (5.632s) 2022-11-23T02:36:08.0226625Z 2022-11-23T02:36:08.0226890Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0226985Z Ran 1 test in 5.633s 2022-11-23T02:36:08.0227039Z 2022-11-23T02:36:08.0227117Z OK 2022-11-23T02:36:08.0227127Z 2022-11-23T02:36:08.0227235Z Generating XML reports... 2022-11-23T02:36:08.0227683Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022534.xml 2022-11-23T02:36:08.0227997Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0228372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0228533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0228910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0229085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0229090Z 2022-11-23T02:36:08.0229183Z Running tests... 2022-11-23T02:36:08.0229449Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0229773Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56758 2022-11-23T02:36:08.0229979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56759 2022-11-23T02:36:08.0230235Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0230612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0230772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0231159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0231334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0231564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0231942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0232102Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0232488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0232664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0232888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0233291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0233687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0233953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0234167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0234401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxazi6fwl 2022-11-23T02:36:08.0234650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxazi6fwl/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0234921Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0235155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_rh0a75 2022-11-23T02:36:08.0235400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_rh0a75/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0235676Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0235760Z ok (5.423s) 2022-11-23T02:36:08.0235766Z 2022-11-23T02:36:08.0236082Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0236177Z Ran 1 test in 5.424s 2022-11-23T02:36:08.0236183Z 2022-11-23T02:36:08.0236259Z OK 2022-11-23T02:36:08.0236264Z 2022-11-23T02:36:08.0236374Z Generating XML reports... 2022-11-23T02:36:08.0236819Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022544.xml 2022-11-23T02:36:08.0237134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0237509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0237669Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0238057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0238233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0238241Z 2022-11-23T02:36:08.0238337Z Running tests... 2022-11-23T02:36:08.0238604Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0239539Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.601s) 2022-11-23T02:36:08.0239546Z 2022-11-23T02:36:08.0239810Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0239905Z Ran 1 test in 0.601s 2022-11-23T02:36:08.0239911Z 2022-11-23T02:36:08.0240001Z OK (skipped=1) 2022-11-23T02:36:08.0240007Z 2022-11-23T02:36:08.0240114Z Generating XML reports... 2022-11-23T02:36:08.0240560Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022554.xml 2022-11-23T02:36:08.0240877Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0241246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0241406Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0241790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0241965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0241971Z 2022-11-23T02:36:08.0242063Z Running tests... 2022-11-23T02:36:08.0242329Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0243288Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.600s) 2022-11-23T02:36:08.0243297Z 2022-11-23T02:36:08.0243566Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0243661Z Ran 1 test in 0.600s 2022-11-23T02:36:08.0243666Z 2022-11-23T02:36:08.0243758Z OK (skipped=1) 2022-11-23T02:36:08.0243763Z 2022-11-23T02:36:08.0243875Z Generating XML reports... 2022-11-23T02:36:08.0244319Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022559.xml 2022-11-23T02:36:08.0244634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0245009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0245231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0245628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0245806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0245812Z 2022-11-23T02:36:08.0245903Z Running tests... 2022-11-23T02:36:08.0246167Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0247123Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.597s) 2022-11-23T02:36:08.0247130Z 2022-11-23T02:36:08.0247392Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0247496Z Ran 1 test in 0.597s 2022-11-23T02:36:08.0247502Z 2022-11-23T02:36:08.0247594Z OK (skipped=1) 2022-11-23T02:36:08.0247599Z 2022-11-23T02:36:08.0247708Z Generating XML reports... 2022-11-23T02:36:08.0248150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022604.xml 2022-11-23T02:36:08.0248466Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0248844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0249005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0249392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0249557Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0249573Z 2022-11-23T02:36:08.0249656Z Running tests... 2022-11-23T02:36:08.0249925Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0250421Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.003s) 2022-11-23T02:36:08.0250428Z 2022-11-23T02:36:08.0250735Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0250831Z Ran 1 test in 0.003s 2022-11-23T02:36:08.0250836Z 2022-11-23T02:36:08.0250927Z OK (skipped=1) 2022-11-23T02:36:08.0250933Z 2022-11-23T02:36:08.0251042Z Generating XML reports... 2022-11-23T02:36:08.0251483Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022609.xml 2022-11-23T02:36:08.0251797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0252234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0252399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0252789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0252963Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0252969Z 2022-11-23T02:36:08.0253060Z Running tests... 2022-11-23T02:36:08.0253326Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0253635Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57232 2022-11-23T02:36:08.0253845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57233 2022-11-23T02:36:08.0254102Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0254534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0254695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0255083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0255260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0255475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0255850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0256012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0256396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0256579Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0256802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0257205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0257600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0257816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0258029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0258418Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:36:08.0258679Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:36:08.0258918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31dqj5ik 2022-11-23T02:36:08.0259168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31dqj5ik/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0259445Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0259829Z /opt/conda/lib/python3.8/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op 2022-11-23T02:36:08.0260085Z warnings.warn("Initializing zero-element tensors is a no-op") 2022-11-23T02:36:08.0260322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6su6n_ru 2022-11-23T02:36:08.0260566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6su6n_ru/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0260845Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0260929Z ok (5.430s) 2022-11-23T02:36:08.0260934Z 2022-11-23T02:36:08.0261253Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0261341Z Ran 1 test in 5.431s 2022-11-23T02:36:08.0261347Z 2022-11-23T02:36:08.0261424Z OK 2022-11-23T02:36:08.0261430Z 2022-11-23T02:36:08.0261537Z Generating XML reports... 2022-11-23T02:36:08.0261986Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022613.xml 2022-11-23T02:36:08.0262302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0262678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0262839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0263227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0263406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0263459Z 2022-11-23T02:36:08.0263555Z Running tests... 2022-11-23T02:36:08.0263825Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0264127Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57442 2022-11-23T02:36:08.0264332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57443 2022-11-23T02:36:08.0264589Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0264962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0265122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0265507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0265689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0265912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0266287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0266449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0266834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0267002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0267224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0267628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0268031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0268244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0268470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:36:08.0268683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0268906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:36:08.0269304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0269699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0269784Z ok (5.430s) 2022-11-23T02:36:08.0269790Z 2022-11-23T02:36:08.0270110Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0270208Z Ran 1 test in 5.430s 2022-11-23T02:36:08.0270213Z 2022-11-23T02:36:08.0270290Z OK 2022-11-23T02:36:08.0270296Z 2022-11-23T02:36:08.0270408Z Generating XML reports... 2022-11-23T02:36:08.0270858Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022622.xml 2022-11-23T02:36:08.0271174Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0271551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0271711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0272099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0272277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0272327Z 2022-11-23T02:36:08.0272427Z Running tests... 2022-11-23T02:36:08.0272687Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0272988Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57647 2022-11-23T02:36:08.0273197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57648 2022-11-23T02:36:08.0273452Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0273829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0273991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0274380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0274562Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0274787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0275165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0275327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0275712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0275892Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0276118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0276522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0276922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0277142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0277367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:36:08.0277581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0277803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:36:08.0278202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0278600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0278684Z ok (5.028s) 2022-11-23T02:36:08.0278690Z 2022-11-23T02:36:08.0278954Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0279096Z Ran 1 test in 5.029s 2022-11-23T02:36:08.0279112Z 2022-11-23T02:36:08.0279180Z OK 2022-11-23T02:36:08.0279186Z 2022-11-23T02:36:08.0279296Z Generating XML reports... 2022-11-23T02:36:08.0279746Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022632.xml 2022-11-23T02:36:08.0280063Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0280438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0280599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0280984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0281161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0281212Z 2022-11-23T02:36:08.0281309Z Running tests... 2022-11-23T02:36:08.0281579Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0282514Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.602s) 2022-11-23T02:36:08.0282521Z 2022-11-23T02:36:08.0282783Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0282880Z Ran 1 test in 0.602s 2022-11-23T02:36:08.0282885Z 2022-11-23T02:36:08.0282977Z OK (skipped=1) 2022-11-23T02:36:08.0282982Z 2022-11-23T02:36:08.0283088Z Generating XML reports... 2022-11-23T02:36:08.0283535Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022642.xml 2022-11-23T02:36:08.0283859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0284235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0284398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0284785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0284961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0284967Z 2022-11-23T02:36:08.0285058Z Running tests... 2022-11-23T02:36:08.0285326Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0285779Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s) 2022-11-23T02:36:08.0285788Z 2022-11-23T02:36:08.0286051Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0286137Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0286151Z 2022-11-23T02:36:08.0286234Z OK (skipped=1) 2022-11-23T02:36:08.0286239Z 2022-11-23T02:36:08.0286347Z Generating XML reports... 2022-11-23T02:36:08.0286789Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022647.xml 2022-11-23T02:36:08.0287106Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0287482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0287643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0288031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0288262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0288269Z 2022-11-23T02:36:08.0288363Z Running tests... 2022-11-23T02:36:08.0288631Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0288944Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57984 2022-11-23T02:36:08.0289152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57985 2022-11-23T02:36:08.0289411Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0289787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0289947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0290333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0290699Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0290927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0291308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0291468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0291851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0292019Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0292246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0292644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0293045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0293257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0293468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0293555Z ok (4.926s) 2022-11-23T02:36:08.0293561Z 2022-11-23T02:36:08.0293828Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0293924Z Ran 1 test in 4.926s 2022-11-23T02:36:08.0293930Z 2022-11-23T02:36:08.0294008Z OK 2022-11-23T02:36:08.0294013Z 2022-11-23T02:36:08.0294120Z Generating XML reports... 2022-11-23T02:36:08.0294565Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022651.xml 2022-11-23T02:36:08.0294881Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0295261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0295420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0295806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0295981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0295987Z 2022-11-23T02:36:08.0296079Z Running tests... 2022-11-23T02:36:08.0296344Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0296580Z test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0296586Z 2022-11-23T02:36:08.0296846Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0296940Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0296949Z 2022-11-23T02:36:08.0297103Z OK (skipped=1) 2022-11-23T02:36:08.0297110Z 2022-11-23T02:36:08.0297211Z Generating XML reports... 2022-11-23T02:36:08.0297657Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022700.xml 2022-11-23T02:36:08.0297970Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0298343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0298505Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0298888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0299062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0299068Z 2022-11-23T02:36:08.0299160Z Running tests... 2022-11-23T02:36:08.0299482Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0299729Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0299736Z 2022-11-23T02:36:08.0300000Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0300099Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0300104Z 2022-11-23T02:36:08.0300195Z OK (skipped=1) 2022-11-23T02:36:08.0300200Z 2022-11-23T02:36:08.0300309Z Generating XML reports... 2022-11-23T02:36:08.0300750Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022704.xml 2022-11-23T02:36:08.0301067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0301442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0301604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0301995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0302172Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0302178Z 2022-11-23T02:36:08.0302270Z Running tests... 2022-11-23T02:36:08.0302532Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0302819Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58319 2022-11-23T02:36:08.0303023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58320 2022-11-23T02:36:08.0303281Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0303655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0303826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0304208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0304384Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0304606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0304980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0305144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0305529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0305704Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0305980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0306387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0306784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0306997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0307213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0307547Z STAGE:2022-11-23 02:27:12 58319:58319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0307823Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0308154Z STAGE:2022-11-23 02:27:12 58320:58320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0308426Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0308824Z STAGE:2022-11-23 02:27:12 58319:58319 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0309182Z STAGE:2022-11-23 02:27:12 58319:58319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0309397Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0309735Z STAGE:2022-11-23 02:27:12 58320:58320 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0310087Z STAGE:2022-11-23 02:27:12 58320:58320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0310309Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0310643Z STAGE:2022-11-23 02:27:12 58319:58319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0310974Z STAGE:2022-11-23 02:27:12 58320:58320 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0311315Z STAGE:2022-11-23 02:27:12 58320:58320 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0311664Z STAGE:2022-11-23 02:27:12 58320:58320 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0312000Z STAGE:2022-11-23 02:27:12 58319:58319 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0312352Z STAGE:2022-11-23 02:27:12 58319:58319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0312438Z ok (5.613s) 2022-11-23T02:36:08.0312444Z 2022-11-23T02:36:08.0312709Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0312805Z Ran 1 test in 5.614s 2022-11-23T02:36:08.0312811Z 2022-11-23T02:36:08.0312886Z OK 2022-11-23T02:36:08.0312892Z 2022-11-23T02:36:08.0312999Z Generating XML reports... 2022-11-23T02:36:08.0313446Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022709.xml 2022-11-23T02:36:08.0313766Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0314140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0314302Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0314690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0314866Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0314872Z 2022-11-23T02:36:08.0314965Z Running tests... 2022-11-23T02:36:08.0315218Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0315470Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:08.0315479Z 2022-11-23T02:36:08.0315809Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0315907Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0315913Z 2022-11-23T02:36:08.0316003Z OK (skipped=1) 2022-11-23T02:36:08.0316008Z 2022-11-23T02:36:08.0316117Z Generating XML reports... 2022-11-23T02:36:08.0316565Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022719.xml 2022-11-23T02:36:08.0316880Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0317257Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0317418Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0317803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0318029Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0318035Z 2022-11-23T02:36:08.0318126Z Running tests... 2022-11-23T02:36:08.0318397Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0318643Z test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0318649Z 2022-11-23T02:36:08.0318910Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0319003Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0319009Z 2022-11-23T02:36:08.0319099Z OK (skipped=1) 2022-11-23T02:36:08.0319105Z 2022-11-23T02:36:08.0319211Z Generating XML reports... 2022-11-23T02:36:08.0319653Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022723.xml 2022-11-23T02:36:08.0319971Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0320352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0320512Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0320886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0321063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0321069Z 2022-11-23T02:36:08.0321163Z Running tests... 2022-11-23T02:36:08.0321428Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0321730Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58667 2022-11-23T02:36:08.0321941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58668 2022-11-23T02:36:08.0322198Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0322573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0322733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0323117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0323292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0323517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0323889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0324048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0324431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0324660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0324891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0325294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0325692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0325905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0326119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0326398Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0326662Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0326796Z ok (5.633s) 2022-11-23T02:36:08.0326805Z 2022-11-23T02:36:08.0327072Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0327170Z Ran 1 test in 5.633s 2022-11-23T02:36:08.0327176Z 2022-11-23T02:36:08.0327252Z OK 2022-11-23T02:36:08.0327258Z 2022-11-23T02:36:08.0327365Z Generating XML reports... 2022-11-23T02:36:08.0327808Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022727.xml 2022-11-23T02:36:08.0328122Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0328498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0328659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0329040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0329222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0329227Z 2022-11-23T02:36:08.0329322Z Running tests... 2022-11-23T02:36:08.0329584Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0330508Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.595s) 2022-11-23T02:36:08.0330515Z 2022-11-23T02:36:08.0330827Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0330923Z Ran 1 test in 0.595s 2022-11-23T02:36:08.0330928Z 2022-11-23T02:36:08.0331020Z OK (skipped=1) 2022-11-23T02:36:08.0331025Z 2022-11-23T02:36:08.0331133Z Generating XML reports... 2022-11-23T02:36:08.0331581Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022737.xml 2022-11-23T02:36:08.0331898Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0332273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0332433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0332818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0332995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0333001Z 2022-11-23T02:36:08.0333084Z Running tests... 2022-11-23T02:36:08.0333346Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0333702Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58943 2022-11-23T02:36:08.0333914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58944 2022-11-23T02:36:08.0334171Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0334548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0334709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0335093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0335268Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0335492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0335868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0336076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0336465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0336639Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0336863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0337261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0337659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0337874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0338105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:36:08.0338319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0338541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:36:08.0338937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0339329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0339406Z ok (5.074s) 2022-11-23T02:36:08.0339420Z 2022-11-23T02:36:08.0339674Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0339770Z Ran 1 test in 5.075s 2022-11-23T02:36:08.0339776Z 2022-11-23T02:36:08.0339852Z OK 2022-11-23T02:36:08.0339858Z 2022-11-23T02:36:08.0339969Z Generating XML reports... 2022-11-23T02:36:08.0340417Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022742.xml 2022-11-23T02:36:08.0340732Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0341106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0341267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0341649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0341822Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0341828Z 2022-11-23T02:36:08.0341921Z Running tests... 2022-11-23T02:36:08.0342186Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0342529Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59148 2022-11-23T02:36:08.0342743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59149 2022-11-23T02:36:08.0342996Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0343373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0343533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0343917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0344092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0344317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0344691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0344889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0345276Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0345456Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0345679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0346078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0346474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0346684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0346894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0347175Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0347449Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0347533Z ok (5.623s) 2022-11-23T02:36:08.0347539Z 2022-11-23T02:36:08.0347802Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0347896Z Ran 1 test in 5.624s 2022-11-23T02:36:08.0347902Z 2022-11-23T02:36:08.0347982Z OK 2022-11-23T02:36:08.0347987Z 2022-11-23T02:36:08.0348095Z Generating XML reports... 2022-11-23T02:36:08.0348538Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022751.xml 2022-11-23T02:36:08.0348853Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0349232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0349400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0349783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0349958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0349963Z 2022-11-23T02:36:08.0350055Z Running tests... 2022-11-23T02:36:08.0350309Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0350600Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59358 2022-11-23T02:36:08.0350805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59359 2022-11-23T02:36:08.0351060Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0351482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0351648Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0352033Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0352209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0352433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0352805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0352964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0353345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0353521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0353790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0354191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0354585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0354799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0355015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0355099Z ok (5.720s) 2022-11-23T02:36:08.0355105Z 2022-11-23T02:36:08.0355368Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0355466Z Ran 1 test in 5.721s 2022-11-23T02:36:08.0355471Z 2022-11-23T02:36:08.0355549Z OK 2022-11-23T02:36:08.0355558Z 2022-11-23T02:36:08.0355657Z Generating XML reports... 2022-11-23T02:36:08.0356100Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022801.xml 2022-11-23T02:36:08.0356415Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0356793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0356955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0357342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0357516Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0357522Z 2022-11-23T02:36:08.0357615Z Running tests... 2022-11-23T02:36:08.0357879Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0358193Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59561 2022-11-23T02:36:08.0358400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59562 2022-11-23T02:36:08.0358656Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0359028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0359190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0359573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0359748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0359971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0360398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0360562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0360951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0361126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0361350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0361741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0362136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0362350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0362622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:36:08.0362836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0363059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:36:08.0363460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0363854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0363940Z ok (5.531s) 2022-11-23T02:36:08.0363946Z 2022-11-23T02:36:08.0364210Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0364308Z Ran 1 test in 5.531s 2022-11-23T02:36:08.0364314Z 2022-11-23T02:36:08.0364391Z OK 2022-11-23T02:36:08.0364399Z 2022-11-23T02:36:08.0364507Z Generating XML reports... 2022-11-23T02:36:08.0364955Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022811.xml 2022-11-23T02:36:08.0365270Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0365649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0365811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0366195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0366371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0366377Z 2022-11-23T02:36:08.0366470Z Running tests... 2022-11-23T02:36:08.0366732Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0367044Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59766 2022-11-23T02:36:08.0367254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59767 2022-11-23T02:36:08.0367497Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0367871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0368034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0368419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0368595Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0368821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0369328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0369493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0369882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0370058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0370282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0370721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0371117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0371330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0371616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T02:36:08.0371831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0372052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T02:36:08.0372455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0372849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T02:36:08.0372934Z ok (4.920s) 2022-11-23T02:36:08.0372940Z 2022-11-23T02:36:08.0373205Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0373300Z Ran 1 test in 4.920s 2022-11-23T02:36:08.0373306Z 2022-11-23T02:36:08.0373382Z OK 2022-11-23T02:36:08.0373388Z 2022-11-23T02:36:08.0373492Z Generating XML reports... 2022-11-23T02:36:08.0373941Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022821.xml 2022-11-23T02:36:08.0374260Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0374637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0374797Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0375184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0375358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0375364Z 2022-11-23T02:36:08.0375455Z Running tests... 2022-11-23T02:36:08.0375719Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0376156Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.003s) 2022-11-23T02:36:08.0376166Z 2022-11-23T02:36:08.0376428Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0376523Z Ran 1 test in 0.003s 2022-11-23T02:36:08.0376529Z 2022-11-23T02:36:08.0376622Z OK (skipped=1) 2022-11-23T02:36:08.0376628Z 2022-11-23T02:36:08.0376737Z Generating XML reports... 2022-11-23T02:36:08.0377182Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022830.xml 2022-11-23T02:36:08.0377497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0377871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0378032Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0378474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0378655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0378661Z 2022-11-23T02:36:08.0378754Z Running tests... 2022-11-23T02:36:08.0379022Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0379247Z test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s) 2022-11-23T02:36:08.0379253Z 2022-11-23T02:36:08.0379506Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0379605Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0379611Z 2022-11-23T02:36:08.0379703Z OK (skipped=1) 2022-11-23T02:36:08.0379708Z 2022-11-23T02:36:08.0379815Z Generating XML reports... 2022-11-23T02:36:08.0380255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022834.xml 2022-11-23T02:36:08.0380634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0381009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0381169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0381552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0381727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0381733Z 2022-11-23T02:36:08.0381824Z Running tests... 2022-11-23T02:36:08.0382087Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0382310Z test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:36:08.0382316Z 2022-11-23T02:36:08.0382578Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0382676Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0382684Z 2022-11-23T02:36:08.0382775Z OK (skipped=1) 2022-11-23T02:36:08.0382780Z 2022-11-23T02:36:08.0382886Z Generating XML reports... 2022-11-23T02:36:08.0383328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022838.xml 2022-11-23T02:36:08.0383641Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0384014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0384175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0384559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0384726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0384741Z 2022-11-23T02:36:08.0384829Z Running tests... 2022-11-23T02:36:08.0385094Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0385346Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:36:08.0385352Z 2022-11-23T02:36:08.0385614Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0385709Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0385715Z 2022-11-23T02:36:08.0385805Z OK (skipped=1) 2022-11-23T02:36:08.0385811Z 2022-11-23T02:36:08.0385920Z Generating XML reports... 2022-11-23T02:36:08.0386359Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022842.xml 2022-11-23T02:36:08.0386672Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0387045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0387257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0387651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0387826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0387832Z 2022-11-23T02:36:08.0387924Z Running tests... 2022-11-23T02:36:08.0388188Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0388435Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.001s) 2022-11-23T02:36:08.0388441Z 2022-11-23T02:36:08.0388702Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0388798Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0388804Z 2022-11-23T02:36:08.0388894Z OK (skipped=1) 2022-11-23T02:36:08.0388899Z 2022-11-23T02:36:08.0389007Z Generating XML reports... 2022-11-23T02:36:08.0389517Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022846.xml 2022-11-23T02:36:08.0389834Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0390198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0390363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0390749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0390926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0390932Z 2022-11-23T02:36:08.0391024Z Running tests... 2022-11-23T02:36:08.0391288Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0391751Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:36:08.0391760Z 2022-11-23T02:36:08.0392023Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0392117Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0392123Z 2022-11-23T02:36:08.0392218Z OK (skipped=1) 2022-11-23T02:36:08.0392224Z 2022-11-23T02:36:08.0392333Z Generating XML reports... 2022-11-23T02:36:08.0392773Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022851.xml 2022-11-23T02:36:08.0393088Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0393461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0393622Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0394007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0394188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0394194Z 2022-11-23T02:36:08.0394286Z Running tests... 2022-11-23T02:36:08.0394548Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0395038Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:36:08.0395045Z 2022-11-23T02:36:08.0395308Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0395403Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0395408Z 2022-11-23T02:36:08.0395499Z OK (skipped=1) 2022-11-23T02:36:08.0395505Z 2022-11-23T02:36:08.0395614Z Generating XML reports... 2022-11-23T02:36:08.0396043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022855.xml 2022-11-23T02:36:08.0396412Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0396791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0396954Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0397341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0397518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0397524Z 2022-11-23T02:36:08.0397618Z Running tests... 2022-11-23T02:36:08.0397882Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0398299Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:08.0398306Z 2022-11-23T02:36:08.0398569Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0398716Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0398721Z 2022-11-23T02:36:08.0398812Z OK (skipped=1) 2022-11-23T02:36:08.0398817Z 2022-11-23T02:36:08.0398926Z Generating XML reports... 2022-11-23T02:36:08.0399369Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022859.xml 2022-11-23T02:36:08.0399683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0400055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0400215Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0400599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0400776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0400786Z 2022-11-23T02:36:08.0400881Z Running tests... 2022-11-23T02:36:08.0401144Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0401542Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:08.0401548Z 2022-11-23T02:36:08.0401810Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0401896Z Ran 1 test in 0.003s 2022-11-23T02:36:08.0401910Z 2022-11-23T02:36:08.0401992Z OK (skipped=1) 2022-11-23T02:36:08.0401998Z 2022-11-23T02:36:08.0402108Z Generating XML reports... 2022-11-23T02:36:08.0402547Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022903.xml 2022-11-23T02:36:08.0402862Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0403238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0403400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0403785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0403964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0403970Z 2022-11-23T02:36:08.0404061Z Running tests... 2022-11-23T02:36:08.0404325Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0404755Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:08.0404761Z 2022-11-23T02:36:08.0405022Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0405116Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0405122Z 2022-11-23T02:36:08.0405213Z OK (skipped=1) 2022-11-23T02:36:08.0405222Z 2022-11-23T02:36:08.0405380Z Generating XML reports... 2022-11-23T02:36:08.0405824Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022907.xml 2022-11-23T02:36:08.0406138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0406511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0406672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0407056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0407232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0407238Z 2022-11-23T02:36:08.0407333Z Running tests... 2022-11-23T02:36:08.0407586Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0408062Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:08.0408069Z 2022-11-23T02:36:08.0408332Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0408429Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0408435Z 2022-11-23T02:36:08.0408525Z OK (skipped=1) 2022-11-23T02:36:08.0408530Z 2022-11-23T02:36:08.0408637Z Generating XML reports... 2022-11-23T02:36:08.0409078Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022911.xml 2022-11-23T02:36:08.0409392Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0409764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0409923Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0410313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0410488Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0410494Z 2022-11-23T02:36:08.0410717Z Running tests... 2022-11-23T02:36:08.0410984Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0411402Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:08.0411408Z 2022-11-23T02:36:08.0411670Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0411764Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0411769Z 2022-11-23T02:36:08.0411859Z OK (skipped=1) 2022-11-23T02:36:08.0411864Z 2022-11-23T02:36:08.0411973Z Generating XML reports... 2022-11-23T02:36:08.0412414Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022915.xml 2022-11-23T02:36:08.0412736Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0413113Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0413276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0413651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0413827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0413832Z 2022-11-23T02:36:08.0413924Z Running tests... 2022-11-23T02:36:08.0414189Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0414506Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60763 2022-11-23T02:36:08.0414785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60764 2022-11-23T02:36:08.0415040Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0415415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0415576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0415959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0416134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0416362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0416736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0416951Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0417336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0417512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0417739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0418140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0418536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0418750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0418963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0419244Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0419518Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0419595Z ok (5.526s) 2022-11-23T02:36:08.0419600Z 2022-11-23T02:36:08.0419864Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0419959Z Ran 1 test in 5.526s 2022-11-23T02:36:08.0419965Z 2022-11-23T02:36:08.0420041Z OK 2022-11-23T02:36:08.0420046Z 2022-11-23T02:36:08.0420155Z Generating XML reports... 2022-11-23T02:36:08.0420595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022919.xml 2022-11-23T02:36:08.0420913Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0421282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0421444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0421833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0422007Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0422013Z 2022-11-23T02:36:08.0422105Z Running tests... 2022-11-23T02:36:08.0422367Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0422687Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60977 2022-11-23T02:36:08.0422898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60978 2022-11-23T02:36:08.0423151Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0423524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0423744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0424140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0424316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0424543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0424908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0425070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0425458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0425633Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0425905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0426307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0426703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0426919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0427134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0427408Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0427680Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0427766Z ok (5.336s) 2022-11-23T02:36:08.0427772Z 2022-11-23T02:36:08.0428033Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0428131Z Ran 1 test in 5.337s 2022-11-23T02:36:08.0428139Z 2022-11-23T02:36:08.0428220Z OK 2022-11-23T02:36:08.0428226Z 2022-11-23T02:36:08.0428334Z Generating XML reports... 2022-11-23T02:36:08.0428775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022928.xml 2022-11-23T02:36:08.0429091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0429463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0429624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0430007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0430181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0430187Z 2022-11-23T02:36:08.0430274Z Running tests... 2022-11-23T02:36:08.0430541Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0430862Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61191 2022-11-23T02:36:08.0431069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61192 2022-11-23T02:36:08.0431323Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0431698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0431859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0432246Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0432421Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0432699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0433078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0433239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0433624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0433800Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0434025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0434426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0434822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0435087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0435301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0435576Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0435853Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0435937Z ok (5.428s) 2022-11-23T02:36:08.0435943Z 2022-11-23T02:36:08.0436197Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0436294Z Ran 1 test in 5.428s 2022-11-23T02:36:08.0436299Z 2022-11-23T02:36:08.0436375Z OK 2022-11-23T02:36:08.0436381Z 2022-11-23T02:36:08.0436489Z Generating XML reports... 2022-11-23T02:36:08.0436933Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022938.xml 2022-11-23T02:36:08.0437256Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0437629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0437791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0438178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0438355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0438361Z 2022-11-23T02:36:08.0438452Z Running tests... 2022-11-23T02:36:08.0438716Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0439032Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61405 2022-11-23T02:36:08.0439237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61406 2022-11-23T02:36:08.0439499Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0439874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0440036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0440421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0440596Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0440819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0441195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0441356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0441786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0441965Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0442189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0442593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0442988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0443203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0443421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0443695Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0444024Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0444110Z ok (5.422s) 2022-11-23T02:36:08.0444116Z 2022-11-23T02:36:08.0444379Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0444475Z Ran 1 test in 5.422s 2022-11-23T02:36:08.0444480Z 2022-11-23T02:36:08.0444557Z OK 2022-11-23T02:36:08.0444562Z 2022-11-23T02:36:08.0444669Z Generating XML reports... 2022-11-23T02:36:08.0445115Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022948.xml 2022-11-23T02:36:08.0445430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0445805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0445969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0446361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0446538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0446544Z 2022-11-23T02:36:08.0446635Z Running tests... 2022-11-23T02:36:08.0446900Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0447831Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.643s) 2022-11-23T02:36:08.0447838Z 2022-11-23T02:36:08.0448100Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0448199Z Ran 1 test in 0.644s 2022-11-23T02:36:08.0448208Z 2022-11-23T02:36:08.0448300Z OK (skipped=1) 2022-11-23T02:36:08.0448307Z 2022-11-23T02:36:08.0448417Z Generating XML reports... 2022-11-23T02:36:08.0448861Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022957.xml 2022-11-23T02:36:08.0449175Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0449550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0449711Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0450086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0450262Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0450268Z 2022-11-23T02:36:08.0450360Z Running tests... 2022-11-23T02:36:08.0450693Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0451009Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:36:08.0451016Z 2022-11-23T02:36:08.0451288Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0451383Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0451388Z 2022-11-23T02:36:08.0451480Z OK (skipped=1) 2022-11-23T02:36:08.0451486Z 2022-11-23T02:36:08.0451593Z Generating XML reports... 2022-11-23T02:36:08.0452035Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023002.xml 2022-11-23T02:36:08.0452350Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0452722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0452883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0453325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0453501Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0453506Z 2022-11-23T02:36:08.0453599Z Running tests... 2022-11-23T02:36:08.0453863Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0454123Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:36:08.0454129Z 2022-11-23T02:36:08.0454389Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0454485Z Ran 1 test in 0.003s 2022-11-23T02:36:08.0454490Z 2022-11-23T02:36:08.0454581Z OK (skipped=1) 2022-11-23T02:36:08.0454586Z 2022-11-23T02:36:08.0454694Z Generating XML reports... 2022-11-23T02:36:08.0455122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023006.xml 2022-11-23T02:36:08.0455447Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0455820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0455981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0456365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0456544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0456550Z 2022-11-23T02:36:08.0456642Z Running tests... 2022-11-23T02:36:08.0456904Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0457202Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:36:08.0457211Z 2022-11-23T02:36:08.0457475Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0457572Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0457577Z 2022-11-23T02:36:08.0457668Z OK (skipped=1) 2022-11-23T02:36:08.0457673Z 2022-11-23T02:36:08.0457783Z Generating XML reports... 2022-11-23T02:36:08.0458223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023010.xml 2022-11-23T02:36:08.0458539Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0458913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0459074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0459462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0459691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0459697Z 2022-11-23T02:36:08.0459791Z Running tests... 2022-11-23T02:36:08.0460058Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0460404Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61881 2022-11-23T02:36:08.0460613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61882 2022-11-23T02:36:08.0460869Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0461233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0461394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0461783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0462008Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0462234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0462613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0462773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0463156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0463332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0463559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0463958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0464361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0464576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0464791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0464876Z ok (4.922s) 2022-11-23T02:36:08.0464882Z 2022-11-23T02:36:08.0465146Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0465244Z Ran 1 test in 4.922s 2022-11-23T02:36:08.0465250Z 2022-11-23T02:36:08.0465327Z OK 2022-11-23T02:36:08.0465332Z 2022-11-23T02:36:08.0465440Z Generating XML reports... 2022-11-23T02:36:08.0465882Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023014.xml 2022-11-23T02:36:08.0466200Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0466577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0466728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0467111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0467286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0467291Z 2022-11-23T02:36:08.0467382Z Running tests... 2022-11-23T02:36:08.0467651Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0467990Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62084 2022-11-23T02:36:08.0468199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62085 2022-11-23T02:36:08.0468503Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0468886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0469048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0469433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0469609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0469837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0470212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0470374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0470811Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0470989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0471214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0471612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0472007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0472221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0472434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0472520Z ok (5.285s) 2022-11-23T02:36:08.0472526Z 2022-11-23T02:36:08.0472788Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0472886Z Ran 1 test in 5.286s 2022-11-23T02:36:08.0472891Z 2022-11-23T02:36:08.0472972Z OK 2022-11-23T02:36:08.0472977Z 2022-11-23T02:36:08.0473085Z Generating XML reports... 2022-11-23T02:36:08.0473527Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023024.xml 2022-11-23T02:36:08.0473844Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0474218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0474381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0474768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0474949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0474958Z 2022-11-23T02:36:08.0475055Z Running tests... 2022-11-23T02:36:08.0475318Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0475582Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:36:08.0475588Z 2022-11-23T02:36:08.0475849Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0475945Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0475951Z 2022-11-23T02:36:08.0476043Z OK (skipped=1) 2022-11-23T02:36:08.0476049Z 2022-11-23T02:36:08.0476158Z Generating XML reports... 2022-11-23T02:36:08.0476600Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023033.xml 2022-11-23T02:36:08.0476918Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0477346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0477514Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0477905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0478072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0478086Z 2022-11-23T02:36:08.0478169Z Running tests... 2022-11-23T02:36:08.0478435Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0478723Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s) 2022-11-23T02:36:08.0478729Z 2022-11-23T02:36:08.0478993Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0479090Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0479096Z 2022-11-23T02:36:08.0479187Z OK (skipped=1) 2022-11-23T02:36:08.0479236Z 2022-11-23T02:36:08.0479351Z Generating XML reports... 2022-11-23T02:36:08.0479799Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023037.xml 2022-11-23T02:36:08.0480117Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0480493Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0480654Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0481042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0481221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0481227Z 2022-11-23T02:36:08.0481322Z Running tests... 2022-11-23T02:36:08.0481584Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0482536Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.590s) 2022-11-23T02:36:08.0482543Z 2022-11-23T02:36:08.0482805Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0482901Z Ran 1 test in 0.591s 2022-11-23T02:36:08.0482907Z 2022-11-23T02:36:08.0482999Z OK (skipped=1) 2022-11-23T02:36:08.0483005Z 2022-11-23T02:36:08.0483113Z Generating XML reports... 2022-11-23T02:36:08.0483556Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023041.xml 2022-11-23T02:36:08.0483870Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0484250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0484412Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0484799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0484966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0484983Z 2022-11-23T02:36:08.0485066Z Running tests... 2022-11-23T02:36:08.0485331Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0485657Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62485 2022-11-23T02:36:08.0485864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62486 2022-11-23T02:36:08.0486173Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0486559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0486721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0487106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0487281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0487505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0487882Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0488045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0488431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0488653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0488876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0489280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0489678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0489896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0490110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0490347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd3u33z_ 2022-11-23T02:36:08.0490648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd3u33z_/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0490887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpya923oex 2022-11-23T02:36:08.0491126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpya923oex/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0491404Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0491678Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0491763Z ok (7.644s) 2022-11-23T02:36:08.0491769Z 2022-11-23T02:36:08.0492035Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0492130Z Ran 1 test in 7.644s 2022-11-23T02:36:08.0492136Z 2022-11-23T02:36:08.0492215Z OK 2022-11-23T02:36:08.0492221Z 2022-11-23T02:36:08.0492329Z Generating XML reports... 2022-11-23T02:36:08.0492771Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023046.xml 2022-11-23T02:36:08.0493091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0493464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0493626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0494012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0494188Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0494193Z 2022-11-23T02:36:08.0494286Z Running tests... 2022-11-23T02:36:08.0494550Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0494864Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62705 2022-11-23T02:36:08.0495130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62706 2022-11-23T02:36:08.0495389Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0495769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0495931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0496315Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0496481Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0496704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0497079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0497294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0497683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0497863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0498087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0498486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0498884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0499098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0499316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0499598Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0499871Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0499956Z ok (5.633s) 2022-11-23T02:36:08.0499962Z 2022-11-23T02:36:08.0500227Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0500321Z Ran 1 test in 5.633s 2022-11-23T02:36:08.0500327Z 2022-11-23T02:36:08.0500404Z OK 2022-11-23T02:36:08.0500409Z 2022-11-23T02:36:08.0500516Z Generating XML reports... 2022-11-23T02:36:08.0500960Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023058.xml 2022-11-23T02:36:08.0501273Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0501649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0501800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0502192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0502370Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0502375Z 2022-11-23T02:36:08.0502472Z Running tests... 2022-11-23T02:36:08.0502739Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0503073Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62919 2022-11-23T02:36:08.0503280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62920 2022-11-23T02:36:08.0503532Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0503908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0504122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0504514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0504688Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0504910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0505288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0505449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0505835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0506014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0506285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0506690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0507086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0507299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0507516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0507795Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0508060Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0508145Z ok (5.749s) 2022-11-23T02:36:08.0508151Z 2022-11-23T02:36:08.0508414Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0508514Z Ran 1 test in 5.749s 2022-11-23T02:36:08.0508522Z 2022-11-23T02:36:08.0508599Z OK 2022-11-23T02:36:08.0508605Z 2022-11-23T02:36:08.0508713Z Generating XML reports... 2022-11-23T02:36:08.0509159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023108.xml 2022-11-23T02:36:08.0509475Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0509850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0510012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0510398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0510574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0510580Z 2022-11-23T02:36:08.0510673Z Running tests... 2022-11-23T02:36:08.0510944Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0511887Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.595s) 2022-11-23T02:36:08.0511894Z 2022-11-23T02:36:08.0512156Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0512253Z Ran 1 test in 0.595s 2022-11-23T02:36:08.0512258Z 2022-11-23T02:36:08.0512350Z OK (skipped=1) 2022-11-23T02:36:08.0512356Z 2022-11-23T02:36:08.0512463Z Generating XML reports... 2022-11-23T02:36:08.0512905Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023118.xml 2022-11-23T02:36:08.0513280Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0513667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0513829Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0514219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0514385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0514401Z 2022-11-23T02:36:08.0514484Z Running tests... 2022-11-23T02:36:08.0514750Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0515720Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.597s) 2022-11-23T02:36:08.0515802Z 2022-11-23T02:36:08.0516059Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0516158Z Ran 1 test in 0.597s 2022-11-23T02:36:08.0516164Z 2022-11-23T02:36:08.0516254Z OK (skipped=1) 2022-11-23T02:36:08.0516260Z 2022-11-23T02:36:08.0516367Z Generating XML reports... 2022-11-23T02:36:08.0516810Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023122.xml 2022-11-23T02:36:08.0517125Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0517501Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0517662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0518055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0518235Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0518241Z 2022-11-23T02:36:08.0518334Z Running tests... 2022-11-23T02:36:08.0518598Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0519589Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75052 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.588s) 2022-11-23T02:36:08.0519596Z 2022-11-23T02:36:08.0519858Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0519957Z Ran 1 test in 0.588s 2022-11-23T02:36:08.0519963Z 2022-11-23T02:36:08.0520057Z OK (skipped=1) 2022-11-23T02:36:08.0520062Z 2022-11-23T02:36:08.0520171Z Generating XML reports... 2022-11-23T02:36:08.0520611Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023128.xml 2022-11-23T02:36:08.0520926Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0521300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0521463Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0521846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0522025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0522030Z 2022-11-23T02:36:08.0522123Z Running tests... 2022-11-23T02:36:08.0522443Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0523461Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75139 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.586s) 2022-11-23T02:36:08.0523468Z 2022-11-23T02:36:08.0523729Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0523824Z Ran 1 test in 0.586s 2022-11-23T02:36:08.0523829Z 2022-11-23T02:36:08.0523921Z OK (skipped=1) 2022-11-23T02:36:08.0523926Z 2022-11-23T02:36:08.0524025Z Generating XML reports... 2022-11-23T02:36:08.0524470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023132.xml 2022-11-23T02:36:08.0524843Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0525221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0525383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0525767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0525945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0525950Z 2022-11-23T02:36:08.0526043Z Running tests... 2022-11-23T02:36:08.0526305Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0527257Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.596s) 2022-11-23T02:36:08.0527267Z 2022-11-23T02:36:08.0527529Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0527624Z Ran 1 test in 0.596s 2022-11-23T02:36:08.0527629Z 2022-11-23T02:36:08.0527721Z OK (skipped=1) 2022-11-23T02:36:08.0527726Z 2022-11-23T02:36:08.0527836Z Generating XML reports... 2022-11-23T02:36:08.0528276Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023137.xml 2022-11-23T02:36:08.0528592Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0528967Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0529130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0529521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0529696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0529702Z 2022-11-23T02:36:08.0529793Z Running tests... 2022-11-23T02:36:08.0530057Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0530312Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0530317Z 2022-11-23T02:36:08.0530704Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0530804Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0530810Z 2022-11-23T02:36:08.0530902Z OK (skipped=1) 2022-11-23T02:36:08.0530908Z 2022-11-23T02:36:08.0531007Z Generating XML reports... 2022-11-23T02:36:08.0531516Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023142.xml 2022-11-23T02:36:08.0531842Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0532217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0532376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0532762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0532939Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0532945Z 2022-11-23T02:36:08.0533037Z Running tests... 2022-11-23T02:36:08.0533302Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0533557Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0533610Z 2022-11-23T02:36:08.0533881Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0533976Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0533981Z 2022-11-23T02:36:08.0534074Z OK (skipped=1) 2022-11-23T02:36:08.0534079Z 2022-11-23T02:36:08.0534187Z Generating XML reports... 2022-11-23T02:36:08.0534629Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023146.xml 2022-11-23T02:36:08.0534944Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0535320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0535481Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0535864Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0536042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0536055Z 2022-11-23T02:36:08.0536147Z Running tests... 2022-11-23T02:36:08.0536412Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0536666Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0536680Z 2022-11-23T02:36:08.0536932Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0537029Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0537034Z 2022-11-23T02:36:08.0537125Z OK (skipped=1) 2022-11-23T02:36:08.0537130Z 2022-11-23T02:36:08.0537237Z Generating XML reports... 2022-11-23T02:36:08.0537677Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023151.xml 2022-11-23T02:36:08.0537989Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0538370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0538531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0538916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0539091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0539097Z 2022-11-23T02:36:08.0539189Z Running tests... 2022-11-23T02:36:08.0539452Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0539707Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0539713Z 2022-11-23T02:36:08.0539974Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0540069Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0540075Z 2022-11-23T02:36:08.0540170Z OK (skipped=1) 2022-11-23T02:36:08.0540175Z 2022-11-23T02:36:08.0540339Z Generating XML reports... 2022-11-23T02:36:08.0540787Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023155.xml 2022-11-23T02:36:08.0541104Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0541480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0541641Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0542027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0542196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0542211Z 2022-11-23T02:36:08.0542294Z Running tests... 2022-11-23T02:36:08.0542555Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0542858Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0542865Z 2022-11-23T02:36:08.0543130Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0543228Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0543233Z 2022-11-23T02:36:08.0543324Z OK (skipped=1) 2022-11-23T02:36:08.0543330Z 2022-11-23T02:36:08.0543439Z Generating XML reports... 2022-11-23T02:36:08.0543883Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023159.xml 2022-11-23T02:36:08.0544197Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0544570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0544730Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0545124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0545301Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0545307Z 2022-11-23T02:36:08.0545401Z Running tests... 2022-11-23T02:36:08.0545665Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0545915Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0545921Z 2022-11-23T02:36:08.0546186Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0546281Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0546287Z 2022-11-23T02:36:08.0546379Z OK (skipped=1) 2022-11-23T02:36:08.0546385Z 2022-11-23T02:36:08.0546493Z Generating XML reports... 2022-11-23T02:36:08.0546933Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023203.xml 2022-11-23T02:36:08.0547255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0547619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0547779Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0548166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0548340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0548346Z 2022-11-23T02:36:08.0548438Z Running tests... 2022-11-23T02:36:08.0548702Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0548960Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0548966Z 2022-11-23T02:36:08.0549226Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0549374Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0549380Z 2022-11-23T02:36:08.0549475Z OK (skipped=1) 2022-11-23T02:36:08.0549481Z 2022-11-23T02:36:08.0549590Z Generating XML reports... 2022-11-23T02:36:08.0550034Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023207.xml 2022-11-23T02:36:08.0550349Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0550723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0550885Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0551269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0551446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0551494Z 2022-11-23T02:36:08.0551590Z Running tests... 2022-11-23T02:36:08.0551862Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0552111Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0552118Z 2022-11-23T02:36:08.0552381Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0552480Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0552485Z 2022-11-23T02:36:08.0552567Z OK (skipped=1) 2022-11-23T02:36:08.0552581Z 2022-11-23T02:36:08.0552679Z Generating XML reports... 2022-11-23T02:36:08.0553119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023211.xml 2022-11-23T02:36:08.0553439Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0553812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0553981Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0554367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0554543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0554548Z 2022-11-23T02:36:08.0554641Z Running tests... 2022-11-23T02:36:08.0554907Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0555147Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0555153Z 2022-11-23T02:36:08.0555414Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0555509Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0555515Z 2022-11-23T02:36:08.0555608Z OK (skipped=1) 2022-11-23T02:36:08.0555613Z 2022-11-23T02:36:08.0555725Z Generating XML reports... 2022-11-23T02:36:08.0556167Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023215.xml 2022-11-23T02:36:08.0556484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0556859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0557022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0557404Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0557581Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0557587Z 2022-11-23T02:36:08.0557681Z Running tests... 2022-11-23T02:36:08.0557947Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0558231Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0558250Z 2022-11-23T02:36:08.0558506Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0558604Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0558609Z 2022-11-23T02:36:08.0558701Z OK (skipped=1) 2022-11-23T02:36:08.0558706Z 2022-11-23T02:36:08.0558814Z Generating XML reports... 2022-11-23T02:36:08.0559259Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023219.xml 2022-11-23T02:36:08.0559580Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0559955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0560117Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0560500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0560733Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0560739Z 2022-11-23T02:36:08.0560831Z Running tests... 2022-11-23T02:36:08.0561099Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0561404Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64123 2022-11-23T02:36:08.0561612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64124 2022-11-23T02:36:08.0561868Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0562242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0562402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0562789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0562966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0563193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0563566Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0563718Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0564100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0564281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0564509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0564912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0565312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0565526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0565738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0566074Z STAGE:2022-11-23 02:32:26 64123:64123 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0566352Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0566687Z STAGE:2022-11-23 02:32:26 64124:64124 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0566962Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0567299Z STAGE:2022-11-23 02:32:26 64124:64124 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0567704Z STAGE:2022-11-23 02:32:26 64123:64123 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0568066Z STAGE:2022-11-23 02:32:26 64124:64124 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0568419Z STAGE:2022-11-23 02:32:26 64123:64123 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0568645Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0568870Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0569209Z STAGE:2022-11-23 02:32:26 64123:64123 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0569550Z STAGE:2022-11-23 02:32:26 64123:64123 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0569910Z STAGE:2022-11-23 02:32:26 64123:64123 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0570288Z STAGE:2022-11-23 02:32:26 64124:64124 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0570682Z STAGE:2022-11-23 02:32:26 64124:64124 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0571026Z STAGE:2022-11-23 02:32:26 64124:64124 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0571824Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:36:08.0571911Z warnings.warn( 2022-11-23T02:36:08.0572690Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1701: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions 2022-11-23T02:36:08.0572788Z warnings.warn( 2022-11-23T02:36:08.0572872Z ok (5.513s) 2022-11-23T02:36:08.0572878Z 2022-11-23T02:36:08.0573148Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0573244Z Ran 1 test in 5.514s 2022-11-23T02:36:08.0573250Z 2022-11-23T02:36:08.0573326Z OK 2022-11-23T02:36:08.0573332Z 2022-11-23T02:36:08.0573441Z Generating XML reports... 2022-11-23T02:36:08.0573885Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023223.xml 2022-11-23T02:36:08.0574200Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0574577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0574738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0575132Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0575308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0575314Z 2022-11-23T02:36:08.0575408Z Running tests... 2022-11-23T02:36:08.0575676Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0575924Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0575930Z 2022-11-23T02:36:08.0576191Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0576287Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0576293Z 2022-11-23T02:36:08.0576388Z OK (skipped=1) 2022-11-23T02:36:08.0576393Z 2022-11-23T02:36:08.0576500Z Generating XML reports... 2022-11-23T02:36:08.0577005Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023233.xml 2022-11-23T02:36:08.0577332Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0577698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0577861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0578245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0578422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0578428Z 2022-11-23T02:36:08.0578520Z Running tests... 2022-11-23T02:36:08.0578786Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0579104Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64405 2022-11-23T02:36:08.0579363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64406 2022-11-23T02:36:08.0579619Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0579996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0580156Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0580542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0580719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0580949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0581325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0581494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0581881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0582058Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0582283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0582685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0583083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0583297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0583510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0583842Z STAGE:2022-11-23 02:32:40 64406:64406 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0584173Z STAGE:2022-11-23 02:32:40 64405:64405 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0584453Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0584733Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0585073Z STAGE:2022-11-23 02:32:40 64406:64406 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0585425Z STAGE:2022-11-23 02:32:40 64406:64406 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0585765Z STAGE:2022-11-23 02:32:40 64405:64405 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0585992Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0586411Z STAGE:2022-11-23 02:32:40 64405:64405 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0586640Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0586978Z STAGE:2022-11-23 02:32:40 64405:64405 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0587313Z STAGE:2022-11-23 02:32:40 64405:64405 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0587664Z STAGE:2022-11-23 02:32:40 64405:64405 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0587993Z STAGE:2022-11-23 02:32:40 64406:64406 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0588331Z STAGE:2022-11-23 02:32:40 64406:64406 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0588683Z STAGE:2022-11-23 02:32:40 64406:64406 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0588815Z ok (5.524s) 2022-11-23T02:36:08.0588824Z 2022-11-23T02:36:08.0589094Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0589191Z Ran 1 test in 5.524s 2022-11-23T02:36:08.0589197Z 2022-11-23T02:36:08.0589273Z OK 2022-11-23T02:36:08.0589278Z 2022-11-23T02:36:08.0589387Z Generating XML reports... 2022-11-23T02:36:08.0589831Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023237.xml 2022-11-23T02:36:08.0590138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0590511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0590674Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0591060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0591241Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0591247Z 2022-11-23T02:36:08.0591343Z Running tests... 2022-11-23T02:36:08.0591610Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0591919Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64621 2022-11-23T02:36:08.0592125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64622 2022-11-23T02:36:08.0592381Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0592759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0592921Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0593311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0593492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0593716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0594090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0594251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0594635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0594811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0595035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0595483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0595890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0596106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0596312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0596591Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0596868Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0596954Z ok (5.530s) 2022-11-23T02:36:08.0596960Z 2022-11-23T02:36:08.0597226Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0597323Z Ran 1 test in 5.530s 2022-11-23T02:36:08.0597328Z 2022-11-23T02:36:08.0597405Z OK 2022-11-23T02:36:08.0597411Z 2022-11-23T02:36:08.0597520Z Generating XML reports... 2022-11-23T02:36:08.0598018Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023246.xml 2022-11-23T02:36:08.0598336Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0598713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0598875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0599262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0599439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0599445Z 2022-11-23T02:36:08.0599537Z Running tests... 2022-11-23T02:36:08.0599804Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0600047Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0600058Z 2022-11-23T02:36:08.0600322Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0600418Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0600424Z 2022-11-23T02:36:08.0600515Z OK (skipped=1) 2022-11-23T02:36:08.0600521Z 2022-11-23T02:36:08.0600629Z Generating XML reports... 2022-11-23T02:36:08.0601060Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023256.xml 2022-11-23T02:36:08.0601374Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0601747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0601912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0602297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0602480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0602486Z 2022-11-23T02:36:08.0602577Z Running tests... 2022-11-23T02:36:08.0602843Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0603144Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64901 2022-11-23T02:36:08.0603352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64902 2022-11-23T02:36:08.0603607Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0603983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0604148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0604591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0604770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0604998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0605380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0605542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0605930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0606104Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0606327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0606734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0607181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0607386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0607600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0607938Z STAGE:2022-11-23 02:33:03 64901:64901 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0608216Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0608547Z STAGE:2022-11-23 02:33:03 64902:64902 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0608821Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0609162Z STAGE:2022-11-23 02:33:03 64902:64902 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0609505Z STAGE:2022-11-23 02:33:03 64901:64901 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0609858Z STAGE:2022-11-23 02:33:03 64902:64902 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0610210Z STAGE:2022-11-23 02:33:03 64901:64901 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0610433Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0610705Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0611039Z STAGE:2022-11-23 02:33:03 64902:64902 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0611369Z STAGE:2022-11-23 02:33:03 64901:64901 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0611706Z STAGE:2022-11-23 02:33:03 64902:64902 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0612065Z STAGE:2022-11-23 02:33:03 64902:64902 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0612403Z STAGE:2022-11-23 02:33:03 64901:64901 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0612755Z STAGE:2022-11-23 02:33:03 64901:64901 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0612842Z ok (5.430s) 2022-11-23T02:36:08.0612847Z 2022-11-23T02:36:08.0613114Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0613210Z Ran 1 test in 5.430s 2022-11-23T02:36:08.0613215Z 2022-11-23T02:36:08.0613292Z OK 2022-11-23T02:36:08.0613297Z 2022-11-23T02:36:08.0613395Z Generating XML reports... 2022-11-23T02:36:08.0613844Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023300.xml 2022-11-23T02:36:08.0614225Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0614606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0614770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0615160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0615337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0615343Z 2022-11-23T02:36:08.0615434Z Running tests... 2022-11-23T02:36:08.0615700Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0616008Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65117 2022-11-23T02:36:08.0616213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65118 2022-11-23T02:36:08.0616521Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0616901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0617062Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0617448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0617626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0617855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0618229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0618391Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0618781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0618961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0619188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0619590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0619977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0620191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0620403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0620738Z STAGE:2022-11-23 02:33:13 65117:65117 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0621022Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0621356Z STAGE:2022-11-23 02:33:13 65118:65118 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0621633Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0621970Z STAGE:2022-11-23 02:33:13 65118:65118 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0622303Z STAGE:2022-11-23 02:33:13 65117:65117 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0622654Z STAGE:2022-11-23 02:33:13 65118:65118 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0623003Z STAGE:2022-11-23 02:33:13 65117:65117 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0623227Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0623505Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0623847Z STAGE:2022-11-23 02:33:13 65118:65118 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0624183Z STAGE:2022-11-23 02:33:13 65118:65118 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0624534Z STAGE:2022-11-23 02:33:13 65118:65118 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0624862Z STAGE:2022-11-23 02:33:13 65117:65117 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0625200Z STAGE:2022-11-23 02:33:13 65117:65117 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0625552Z STAGE:2022-11-23 02:33:13 65117:65117 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0625639Z ok (5.523s) 2022-11-23T02:36:08.0625645Z 2022-11-23T02:36:08.0625914Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0626061Z Ran 1 test in 5.524s 2022-11-23T02:36:08.0626066Z 2022-11-23T02:36:08.0626134Z OK 2022-11-23T02:36:08.0626150Z 2022-11-23T02:36:08.0626248Z Generating XML reports... 2022-11-23T02:36:08.0626694Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023310.xml 2022-11-23T02:36:08.0627009Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0627386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0627550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0627937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0628114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0628124Z 2022-11-23T02:36:08.0628219Z Running tests... 2022-11-23T02:36:08.0628487Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0628738Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0628744Z 2022-11-23T02:36:08.0629007Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0629104Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0629110Z 2022-11-23T02:36:08.0629201Z OK (skipped=1) 2022-11-23T02:36:08.0629207Z 2022-11-23T02:36:08.0629315Z Generating XML reports... 2022-11-23T02:36:08.0629757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023320.xml 2022-11-23T02:36:08.0630073Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0630450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0630617Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0631004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0631181Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0631187Z 2022-11-23T02:36:08.0631280Z Running tests... 2022-11-23T02:36:08.0631534Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0631770Z test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:08.0631786Z 2022-11-23T02:36:08.0632039Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0632138Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0632143Z 2022-11-23T02:36:08.0632237Z OK (skipped=1) 2022-11-23T02:36:08.0632243Z 2022-11-23T02:36:08.0632351Z Generating XML reports... 2022-11-23T02:36:08.0632846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023324.xml 2022-11-23T02:36:08.0633168Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0633543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0633707Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0634090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0634267Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0634273Z 2022-11-23T02:36:08.0634367Z Running tests... 2022-11-23T02:36:08.0634631Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0634881Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0634936Z 2022-11-23T02:36:08.0635201Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0635301Z Ran 1 test in 0.003s 2022-11-23T02:36:08.0635307Z 2022-11-23T02:36:08.0635399Z OK (skipped=1) 2022-11-23T02:36:08.0635404Z 2022-11-23T02:36:08.0635513Z Generating XML reports... 2022-11-23T02:36:08.0635958Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023328.xml 2022-11-23T02:36:08.0636274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0636649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0636811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0637185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0637369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0637384Z 2022-11-23T02:36:08.0637468Z Running tests... 2022-11-23T02:36:08.0637731Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0637980Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0637986Z 2022-11-23T02:36:08.0638248Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0638345Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0638350Z 2022-11-23T02:36:08.0638442Z OK (skipped=1) 2022-11-23T02:36:08.0638448Z 2022-11-23T02:36:08.0638557Z Generating XML reports... 2022-11-23T02:36:08.0638996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023332.xml 2022-11-23T02:36:08.0639312Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0639693Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0639857Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0640242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0640419Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0640425Z 2022-11-23T02:36:08.0640519Z Running tests... 2022-11-23T02:36:08.0640783Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0641082Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65597 2022-11-23T02:36:08.0641288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65598 2022-11-23T02:36:08.0641602Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0641980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0642139Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0642523Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0642690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0642917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0643290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0643453Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0643839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0644062Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0644286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0644688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0645084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0645299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0645515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0645849Z STAGE:2022-11-23 02:33:39 65597:65597 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0646136Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0646466Z STAGE:2022-11-23 02:33:39 65598:65598 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0646739Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0647080Z STAGE:2022-11-23 02:33:39 65597:65597 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0647434Z STAGE:2022-11-23 02:33:39 65597:65597 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0647659Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0647996Z STAGE:2022-11-23 02:33:40 65598:65598 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0648345Z STAGE:2022-11-23 02:33:40 65598:65598 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0648577Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0648911Z STAGE:2022-11-23 02:33:40 65597:65597 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0649246Z STAGE:2022-11-23 02:33:40 65597:65597 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0649585Z STAGE:2022-11-23 02:33:40 65597:65597 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0649915Z STAGE:2022-11-23 02:33:40 65598:65598 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0650252Z STAGE:2022-11-23 02:33:40 65598:65598 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0650731Z STAGE:2022-11-23 02:33:40 65598:65598 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0650819Z ok (5.523s) 2022-11-23T02:36:08.0650825Z 2022-11-23T02:36:08.0651093Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0651195Z Ran 1 test in 5.523s 2022-11-23T02:36:08.0651274Z 2022-11-23T02:36:08.0651353Z OK 2022-11-23T02:36:08.0651359Z 2022-11-23T02:36:08.0651468Z Generating XML reports... 2022-11-23T02:36:08.0651918Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023336.xml 2022-11-23T02:36:08.0652232Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0652607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0652769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0653155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0653332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0653338Z 2022-11-23T02:36:08.0653585Z Running tests... 2022-11-23T02:36:08.0653861Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0654172Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65813 2022-11-23T02:36:08.0654379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65814 2022-11-23T02:36:08.0654634Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0655008Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0655170Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0655543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0655719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0655952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0656324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0656485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0656866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0657043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0657270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0657669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0658066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0658285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0658496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0658830Z STAGE:2022-11-23 02:33:49 65813:65813 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0659104Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0659435Z STAGE:2022-11-23 02:33:49 65814:65814 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0659709Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0660046Z STAGE:2022-11-23 02:33:49 65813:65813 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0660397Z STAGE:2022-11-23 02:33:49 65813:65813 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0660670Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0661014Z STAGE:2022-11-23 02:33:49 65814:65814 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0661365Z STAGE:2022-11-23 02:33:49 65814:65814 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0661586Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0661920Z STAGE:2022-11-23 02:33:49 65813:65813 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0662245Z STAGE:2022-11-23 02:33:49 65813:65813 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0662594Z STAGE:2022-11-23 02:33:49 65813:65813 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0662925Z STAGE:2022-11-23 02:33:49 65814:65814 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0663314Z STAGE:2022-11-23 02:33:49 65814:65814 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0663662Z STAGE:2022-11-23 02:33:49 65814:65814 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0663747Z ok (5.431s) 2022-11-23T02:36:08.0663753Z 2022-11-23T02:36:08.0664016Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0664113Z Ran 1 test in 5.431s 2022-11-23T02:36:08.0664119Z 2022-11-23T02:36:08.0664196Z OK 2022-11-23T02:36:08.0664202Z 2022-11-23T02:36:08.0664310Z Generating XML reports... 2022-11-23T02:36:08.0664750Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023346.xml 2022-11-23T02:36:08.0665061Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0665436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0665605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0665989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0666167Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0666172Z 2022-11-23T02:36:08.0666265Z Running tests... 2022-11-23T02:36:08.0666528Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0666781Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.001s) 2022-11-23T02:36:08.0666787Z 2022-11-23T02:36:08.0667047Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0667142Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0667148Z 2022-11-23T02:36:08.0667239Z OK (skipped=1) 2022-11-23T02:36:08.0667245Z 2022-11-23T02:36:08.0667347Z Generating XML reports... 2022-11-23T02:36:08.0667793Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023356.xml 2022-11-23T02:36:08.0668112Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0668484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0668645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0669027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0669203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0669209Z 2022-11-23T02:36:08.0669305Z Running tests... 2022-11-23T02:36:08.0669573Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0669870Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s) 2022-11-23T02:36:08.0669880Z 2022-11-23T02:36:08.0670148Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0670243Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0670249Z 2022-11-23T02:36:08.0670342Z OK (skipped=1) 2022-11-23T02:36:08.0670347Z 2022-11-23T02:36:08.0670455Z Generating XML reports... 2022-11-23T02:36:08.0670897Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023400.xml 2022-11-23T02:36:08.0671210Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0671582Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0671745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0672130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0672354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0672360Z 2022-11-23T02:36:08.0672453Z Running tests... 2022-11-23T02:36:08.0672720Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0673104Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s) 2022-11-23T02:36:08.0673118Z 2022-11-23T02:36:08.0673373Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0673469Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0673474Z 2022-11-23T02:36:08.0673566Z OK (skipped=1) 2022-11-23T02:36:08.0673572Z 2022-11-23T02:36:08.0673680Z Generating XML reports... 2022-11-23T02:36:08.0674125Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023404.xml 2022-11-23T02:36:08.0674442Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0674820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0674982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0675367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0675544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0675550Z 2022-11-23T02:36:08.0675643Z Running tests... 2022-11-23T02:36:08.0675908Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0676159Z test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:36:08.0676165Z 2022-11-23T02:36:08.0676426Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0676524Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0676532Z 2022-11-23T02:36:08.0676624Z OK (skipped=1) 2022-11-23T02:36:08.0676629Z 2022-11-23T02:36:08.0676738Z Generating XML reports... 2022-11-23T02:36:08.0677182Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023408.xml 2022-11-23T02:36:08.0677496Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0677872Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0678034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0678418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0678584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0678600Z 2022-11-23T02:36:08.0678686Z Running tests... 2022-11-23T02:36:08.0678999Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0679274Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.001s) 2022-11-23T02:36:08.0679280Z 2022-11-23T02:36:08.0679545Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0679641Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0679647Z 2022-11-23T02:36:08.0679738Z OK (skipped=1) 2022-11-23T02:36:08.0679743Z 2022-11-23T02:36:08.0679854Z Generating XML reports... 2022-11-23T02:36:08.0680296Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023412.xml 2022-11-23T02:36:08.0680608Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0680985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0681196Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0681586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0681763Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0681769Z 2022-11-23T02:36:08.0681863Z Running tests... 2022-11-23T02:36:08.0682128Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0682428Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s) 2022-11-23T02:36:08.0682434Z 2022-11-23T02:36:08.0682695Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0682789Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0682795Z 2022-11-23T02:36:08.0682888Z OK (skipped=1) 2022-11-23T02:36:08.0682893Z 2022-11-23T02:36:08.0683005Z Generating XML reports... 2022-11-23T02:36:08.0683451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023416.xml 2022-11-23T02:36:08.0683766Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0684143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0684295Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0684679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0684856Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0684862Z 2022-11-23T02:36:08.0684953Z Running tests... 2022-11-23T02:36:08.0685216Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0685513Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s) 2022-11-23T02:36:08.0685522Z 2022-11-23T02:36:08.0685786Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0685882Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0685887Z 2022-11-23T02:36:08.0685979Z OK (skipped=1) 2022-11-23T02:36:08.0685984Z 2022-11-23T02:36:08.0686093Z Generating XML reports... 2022-11-23T02:36:08.0686535Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023420.xml 2022-11-23T02:36:08.0686849Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0687223Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0687383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0687823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0688004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0688010Z 2022-11-23T02:36:08.0688108Z Running tests... 2022-11-23T02:36:08.0688376Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0688652Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:36:08.0688658Z 2022-11-23T02:36:08.0688920Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0689015Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0689021Z 2022-11-23T02:36:08.0689114Z OK (skipped=1) 2022-11-23T02:36:08.0689119Z 2022-11-23T02:36:08.0689228Z Generating XML reports... 2022-11-23T02:36:08.0689656Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023424.xml 2022-11-23T02:36:08.0690020Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0690398Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0690634Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0691196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0691556Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0691572Z 2022-11-23T02:36:08.0691770Z Running tests... 2022-11-23T02:36:08.0692197Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0692499Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66557 2022-11-23T02:36:08.0692707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66558 2022-11-23T02:36:08.0692973Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0693350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0693517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0693900Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0694076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0694301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0694673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0694834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0695226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0695406Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0695631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0696031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0696424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0696629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0696846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0697121Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0697468Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0697557Z ok (5.420s) 2022-11-23T02:36:08.0697563Z 2022-11-23T02:36:08.0697831Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0697927Z Ran 1 test in 5.421s 2022-11-23T02:36:08.0697932Z 2022-11-23T02:36:08.0698011Z OK 2022-11-23T02:36:08.0698016Z 2022-11-23T02:36:08.0698124Z Generating XML reports... 2022-11-23T02:36:08.0698567Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023428.xml 2022-11-23T02:36:08.0698880Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0699253Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0699414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0699858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0700036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0700041Z 2022-11-23T02:36:08.0700133Z Running tests... 2022-11-23T02:36:08.0700401Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0700727Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66767 2022-11-23T02:36:08.0700933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66768 2022-11-23T02:36:08.0701187Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0701562Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0701720Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0702102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0702278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0702502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0702874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0703036Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0703423Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0703600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0703825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0704236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0704634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0704849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0705067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0705401Z STAGE:2022-11-23 02:34:41 66768:66768 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0705730Z STAGE:2022-11-23 02:34:41 66767:66767 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0706009Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0706283Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0706676Z STAGE:2022-11-23 02:34:41 66767:66767 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0707035Z STAGE:2022-11-23 02:34:41 66767:66767 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0707259Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0707596Z STAGE:2022-11-23 02:34:41 66768:66768 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0707948Z STAGE:2022-11-23 02:34:41 66768:66768 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0708172Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 2 (function operator()) 2022-11-23T02:36:08.0708248Z ok (5.669s) 2022-11-23T02:36:08.0708254Z 2022-11-23T02:36:08.0708519Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0708615Z Ran 1 test in 5.669s 2022-11-23T02:36:08.0708665Z 2022-11-23T02:36:08.0708746Z OK 2022-11-23T02:36:08.0708752Z 2022-11-23T02:36:08.0708860Z Generating XML reports... 2022-11-23T02:36:08.0709305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023438.xml 2022-11-23T02:36:08.0709619Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0709992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0710154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0710541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0710719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0710724Z 2022-11-23T02:36:08.0710816Z Running tests... 2022-11-23T02:36:08.0711088Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0711408Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66983 2022-11-23T02:36:08.0711615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66984 2022-11-23T02:36:08.0711873Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0712252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0712414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0712798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0712975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0713204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0713577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0713729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0714112Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0714290Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0714512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0714912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0715309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0715577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0715794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0716131Z STAGE:2022-11-23 02:34:51 66983:66983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0716410Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0716743Z STAGE:2022-11-23 02:34:51 66984:66984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up 2022-11-23T02:36:08.0717018Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0717355Z STAGE:2022-11-23 02:34:51 66983:66983 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0717704Z STAGE:2022-11-23 02:34:51 66983:66983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0717932Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0718319Z STAGE:2022-11-23 02:34:51 66984:66984 ActivityProfilerController.cpp:306] Completed Stage: Collection 2022-11-23T02:36:08.0718670Z STAGE:2022-11-23 02:34:51 66984:66984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing 2022-11-23T02:36:08.0718893Z [W collection.cpp:774] Warning: ROCTracer produced duplicate flow start: 1 (function operator()) 2022-11-23T02:36:08.0718979Z ok (6.055s) 2022-11-23T02:36:08.0718985Z 2022-11-23T02:36:08.0719249Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0719348Z Ran 1 test in 6.056s 2022-11-23T02:36:08.0719353Z 2022-11-23T02:36:08.0719431Z OK 2022-11-23T02:36:08.0719436Z 2022-11-23T02:36:08.0719535Z Generating XML reports... 2022-11-23T02:36:08.0719976Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023448.xml 2022-11-23T02:36:08.0720298Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0720673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0720836Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0721222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0721399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0721405Z 2022-11-23T02:36:08.0721496Z Running tests... 2022-11-23T02:36:08.0721760Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0722032Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s) 2022-11-23T02:36:08.0722039Z 2022-11-23T02:36:08.0722300Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0722402Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0722407Z 2022-11-23T02:36:08.0722501Z OK (skipped=1) 2022-11-23T02:36:08.0722506Z 2022-11-23T02:36:08.0722615Z Generating XML reports... 2022-11-23T02:36:08.0723058Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023458.xml 2022-11-23T02:36:08.0723372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0723746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0723909Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0724293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0724470Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0724480Z 2022-11-23T02:36:08.0724573Z Running tests... 2022-11-23T02:36:08.0724901Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0725154Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:36:08.0725171Z 2022-11-23T02:36:08.0725425Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0725522Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0725528Z 2022-11-23T02:36:08.0725619Z OK (skipped=1) 2022-11-23T02:36:08.0725624Z 2022-11-23T02:36:08.0725733Z Generating XML reports... 2022-11-23T02:36:08.0726178Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023502.xml 2022-11-23T02:36:08.0726492Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0726870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0727082Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0727470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0727646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0727652Z 2022-11-23T02:36:08.0727744Z Running tests... 2022-11-23T02:36:08.0728007Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0728291Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.001s) 2022-11-23T02:36:08.0728297Z 2022-11-23T02:36:08.0728558Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0728656Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0728662Z 2022-11-23T02:36:08.0728753Z OK (skipped=1) 2022-11-23T02:36:08.0728758Z 2022-11-23T02:36:08.0728870Z Generating XML reports... 2022-11-23T02:36:08.0729316Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023506.xml 2022-11-23T02:36:08.0729629Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0730004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0730167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0730608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0730786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0730792Z 2022-11-23T02:36:08.0730875Z Running tests... 2022-11-23T02:36:08.0731141Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0731427Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s) 2022-11-23T02:36:08.0731436Z 2022-11-23T02:36:08.0731697Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0731790Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0731796Z 2022-11-23T02:36:08.0731889Z OK (skipped=1) 2022-11-23T02:36:08.0731895Z 2022-11-23T02:36:08.0732002Z Generating XML reports... 2022-11-23T02:36:08.0732442Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023510.xml 2022-11-23T02:36:08.0732759Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0733136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0733297Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0733741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0733923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0733929Z 2022-11-23T02:36:08.0734019Z Running tests... 2022-11-23T02:36:08.0734286Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0734555Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.001s) 2022-11-23T02:36:08.0734561Z 2022-11-23T02:36:08.0734822Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0734918Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0734923Z 2022-11-23T02:36:08.0735014Z OK (skipped=1) 2022-11-23T02:36:08.0735020Z 2022-11-23T02:36:08.0735132Z Generating XML reports... 2022-11-23T02:36:08.0735573Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023514.xml 2022-11-23T02:36:08.0735946Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0736319Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0736472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0736859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0737036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0737042Z 2022-11-23T02:36:08.0737134Z Running tests... 2022-11-23T02:36:08.0737400Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0737680Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s) 2022-11-23T02:36:08.0737686Z 2022-11-23T02:36:08.0737947Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0738052Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0738058Z 2022-11-23T02:36:08.0738148Z OK (skipped=1) 2022-11-23T02:36:08.0738153Z 2022-11-23T02:36:08.0738263Z Generating XML reports... 2022-11-23T02:36:08.0738702Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023518.xml 2022-11-23T02:36:08.0739018Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0739392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0739552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0739937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0740114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0740123Z 2022-11-23T02:36:08.0740221Z Running tests... 2022-11-23T02:36:08.0740483Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0740794Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67595 2022-11-23T02:36:08.0740999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67596 2022-11-23T02:36:08.0741256Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0741629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0741791Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0742164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0742395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0742623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0742998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0743162Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0743549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0743725Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0743947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0744350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0744752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0745038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0745253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0745534Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0745808Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0745894Z ok (7.532s) 2022-11-23T02:36:08.0745900Z 2022-11-23T02:36:08.0746165Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0746261Z Ran 1 test in 7.533s 2022-11-23T02:36:08.0746267Z 2022-11-23T02:36:08.0746344Z OK 2022-11-23T02:36:08.0746350Z 2022-11-23T02:36:08.0746459Z Generating XML reports... 2022-11-23T02:36:08.0746900Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023522.xml 2022-11-23T02:36:08.0747220Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0747586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0747750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0748137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0748315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0748321Z 2022-11-23T02:36:08.0748414Z Running tests... 2022-11-23T02:36:08.0748678Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0748938Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s) 2022-11-23T02:36:08.0748947Z 2022-11-23T02:36:08.0749210Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0749307Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0749312Z 2022-11-23T02:36:08.0749403Z OK (skipped=1) 2022-11-23T02:36:08.0749408Z 2022-11-23T02:36:08.0749516Z Generating XML reports... 2022-11-23T02:36:08.0749958Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023534.xml 2022-11-23T02:36:08.0750272Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0750645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0750807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0751193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0751369Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0751426Z 2022-11-23T02:36:08.0751523Z Running tests... 2022-11-23T02:36:08.0751790Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0752090Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67881 2022-11-23T02:36:08.0752299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67882 2022-11-23T02:36:08.0752555Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0752935Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0753088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0753473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0753707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0753935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0754309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0754471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0754855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0755031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0755259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0755659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0756062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0756277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0756495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0756730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsl6f63f6 2022-11-23T02:36:08.0756979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsl6f63f6/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0757257Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0757495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpj81s66 2022-11-23T02:36:08.0757745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpj81s66/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0758027Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0758113Z ok (5.415s) 2022-11-23T02:36:08.0758119Z 2022-11-23T02:36:08.0758384Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0758472Z Ran 1 test in 5.415s 2022-11-23T02:36:08.0758487Z 2022-11-23T02:36:08.0758554Z OK 2022-11-23T02:36:08.0758560Z 2022-11-23T02:36:08.0758667Z Generating XML reports... 2022-11-23T02:36:08.0759107Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023538.xml 2022-11-23T02:36:08.0759426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0759802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0759965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0760400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0760583Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0760589Z 2022-11-23T02:36:08.0760682Z Running tests... 2022-11-23T02:36:08.0760949Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0761287Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68091 2022-11-23T02:36:08.0761494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68092 2022-11-23T02:36:08.0761748Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:36:08.0762123Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0762286Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0762726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0762903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0763129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:36:08.0763504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0763666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0764049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0764227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0764442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:36:08.0764848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0765243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:36:08.0765462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:36:08.0765677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:36:08.0765913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw02n1y5r 2022-11-23T02:36:08.0766160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw02n1y5r/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0766436Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0766675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprp4z870n 2022-11-23T02:36:08.0766927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprp4z870n/_remote_module_non_scriptable.py 2022-11-23T02:36:08.0767202Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:36:08.0767971Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:36:08.0768784Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:36:08.0768877Z ok (7.517s) 2022-11-23T02:36:08.0768883Z 2022-11-23T02:36:08.0769154Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0769252Z Ran 1 test in 7.517s 2022-11-23T02:36:08.0769258Z 2022-11-23T02:36:08.0769336Z OK 2022-11-23T02:36:08.0769342Z 2022-11-23T02:36:08.0769449Z Generating XML reports... 2022-11-23T02:36:08.0769897Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023547.xml 2022-11-23T02:36:08.0770212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0770745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0771108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0771587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0771768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0771774Z 2022-11-23T02:36:08.0771866Z Running tests... 2022-11-23T02:36:08.0772130Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0772597Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s) 2022-11-23T02:36:08.0772604Z 2022-11-23T02:36:08.0772868Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0772965Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0772971Z 2022-11-23T02:36:08.0773067Z OK (skipped=1) 2022-11-23T02:36:08.0773072Z 2022-11-23T02:36:08.0773184Z Generating XML reports... 2022-11-23T02:36:08.0773626Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023559.xml 2022-11-23T02:36:08.0773930Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-11-23T02:36:08.0774306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:36:08.0774467Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:36:08.0774850Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:36:08.0775024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:36:08.0775030Z 2022-11-23T02:36:08.0775122Z Running tests... 2022-11-23T02:36:08.0775386Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0775863Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s) 2022-11-23T02:36:08.0775870Z 2022-11-23T02:36:08.0776135Z ---------------------------------------------------------------------- 2022-11-23T02:36:08.0776229Z Ran 1 test in 0.002s 2022-11-23T02:36:08.0776235Z 2022-11-23T02:36:08.0776325Z OK (skipped=1) 2022-11-23T02:36:08.0776330Z 2022-11-23T02:36:08.0776438Z Generating XML reports... 2022-11-23T02:36:08.0776879Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123023603.xml 2022-11-23T02:36:08.0776885Z 2022-11-23T02:36:08.0777388Z ##[endgroup] 2022-11-23T02:36:08.0777864Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_distributed_spawn_7dv6wrqt) 2022-11-23T02:36:08.0777870Z 2022-11-23T02:36:08.0777988Z Shard 2: gloo should be run in 1 2022-11-23T02:36:08.0778307Z Running distributed/test_store ... [2022-11-23 02:36:07.846016] 2022-11-23T02:36:08.0778816Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:36:07.846312] 2022-11-23T02:38:36.2228380Z 2022-11-23T02:38:36.2233073Z Expand the folded group to see the log file of distributed/test_store 2022-11-23T02:38:36.2235196Z ##[group]PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_store_p4nhgkfz) 2022-11-23T02:38:36.2237175Z , <__main__.FileStoreTest testMethod=test_init_pg_and_rpc_with_same_file>, <__main__.FileStoreTest testMethod=test_refcount>, <__main__.FileStoreTest testMethod=test_set_get>]> 2022-11-23T02:38:36.2238819Z test_compare_set (__main__.FileStoreTest) 2022-11-23T02:38:36.2240329Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) 2022-11-23T02:38:36.2241448Z test_refcount (__main__.FileStoreTest) 2022-11-23T02:38:36.2242546Z test_set_get (__main__.FileStoreTest) 2022-11-23T02:38:36.2244025Z , <__main__.HashStoreTest testMethod=test_set_get>]> 2022-11-23T02:38:36.2245793Z test_compare_set (__main__.HashStoreTest) 2022-11-23T02:38:36.2246824Z test_set_get (__main__.HashStoreTest) 2022-11-23T02:38:36.2248464Z , <__main__.PrefixFileStoreTest testMethod=test_set_get>]> 2022-11-23T02:38:36.2250195Z test_compare_set (__main__.PrefixFileStoreTest) 2022-11-23T02:38:36.2251631Z test_set_get (__main__.PrefixFileStoreTest) 2022-11-23T02:38:36.2252980Z ]> 2022-11-23T02:38:36.2254162Z test_get_underlying_store (__main__.PrefixStoreTest) 2022-11-23T02:38:36.2255774Z , <__main__.PrefixTCPStoreTest testMethod=test_set_get>]> 2022-11-23T02:38:36.2257384Z test_compare_set (__main__.PrefixTCPStoreTest) 2022-11-23T02:38:36.2258625Z test_set_get (__main__.PrefixTCPStoreTest) 2022-11-23T02:38:36.2259933Z ]> 2022-11-23T02:38:36.2261220Z test_set_get (__main__.PythonStoreTest) 2022-11-23T02:38:36.2263022Z ]> 2022-11-23T02:38:36.2264335Z test_nominal (__main__.RendezvousEnvTest) 2022-11-23T02:38:36.2265930Z , <__main__.RendezvousFileTest testMethod=test_nominal>]> 2022-11-23T02:38:36.2267516Z test_common_errors (__main__.RendezvousFileTest) 2022-11-23T02:38:36.2268700Z test_nominal (__main__.RendezvousFileTest) 2022-11-23T02:38:36.2270929Z , <__main__.RendezvousTCPTest testMethod=test_dns_timeout>, <__main__.RendezvousTCPTest testMethod=test_nominal>, <__main__.RendezvousTCPTest testMethod=test_tcp_store_timeout_set>]> 2022-11-23T02:38:36.2272951Z test_common_errors (__main__.RendezvousTCPTest) 2022-11-23T02:38:36.2274025Z test_dns_timeout (__main__.RendezvousTCPTest) 2022-11-23T02:38:36.2275033Z test_nominal (__main__.RendezvousTCPTest) 2022-11-23T02:38:36.2276141Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) 2022-11-23T02:38:36.2277769Z , <__main__.RendezvousTest testMethod=test_url_with_node_params>]> 2022-11-23T02:38:36.2279029Z test_unknown_handler (__main__.RendezvousTest) 2022-11-23T02:38:36.2279901Z test_url_with_node_params (__main__.RendezvousTest) 2022-11-23T02:38:36.2282813Z , <__main__.TCPStoreTest testMethod=test_compare_set>, <__main__.TCPStoreTest testMethod=test_init_pg_and_rpc_with_same_socket>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_fixed_world_size>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_nonfixed_world_size>, <__main__.TCPStoreTest testMethod=test_multitenancy>, <__main__.TCPStoreTest testMethod=test_numkeys_delkeys>, <__main__.TCPStoreTest testMethod=test_set_get>]> 2022-11-23T02:38:36.2285350Z test_address_already_in_use (__main__.TCPStoreTest) 2022-11-23T02:38:36.2286187Z test_compare_set (__main__.TCPStoreTest) 2022-11-23T02:38:36.2287083Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) 2022-11-23T02:38:36.2288057Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) 2022-11-23T02:38:36.2289044Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) 2022-11-23T02:38:36.2289925Z test_multitenancy (__main__.TCPStoreTest) 2022-11-23T02:38:36.2291092Z test_numkeys_delkeys (__main__.TCPStoreTest) 2022-11-23T02:38:36.2291905Z test_set_get (__main__.TCPStoreTest) 2022-11-23T02:38:36.2293543Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2295247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2296422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2297976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2299208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2299793Z 2022-11-23T02:38:36.2300040Z Running tests... 2022-11-23T02:38:36.2301126Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2302131Z test_compare_set (__main__.FileStoreTest) ... ok (0.579s) 2022-11-23T02:38:36.2302674Z 2022-11-23T02:38:36.2303392Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2304214Z Ran 1 test in 0.579s 2022-11-23T02:38:36.2304604Z 2022-11-23T02:38:36.2304787Z OK 2022-11-23T02:38:36.2305103Z 2022-11-23T02:38:36.2305391Z Generating XML reports... 2022-11-23T02:38:36.2306879Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123023611.xml 2022-11-23T02:38:36.2308495Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2310161Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2311324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2312888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2314086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2314668Z 2022-11-23T02:38:36.2314919Z Running tests... 2022-11-23T02:38:36.2316002Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2317729Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) ... libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:38:36.2319118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:38:36.2320881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:38:36.2321891Z ok (0.647s) 2022-11-23T02:38:36.2322242Z 2022-11-23T02:38:36.2322939Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2323761Z Ran 1 test in 0.647s 2022-11-23T02:38:36.2324148Z 2022-11-23T02:38:36.2324348Z OK 2022-11-23T02:38:36.2324657Z 2022-11-23T02:38:36.2324948Z Generating XML reports... 2022-11-23T02:38:36.2326618Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123023615.xml 2022-11-23T02:38:36.2328257Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2329918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2331192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2332766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2333985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2334564Z 2022-11-23T02:38:36.2334807Z Running tests... 2022-11-23T02:38:36.2335896Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2336874Z test_refcount (__main__.FileStoreTest) ... ok (0.577s) 2022-11-23T02:38:36.2337386Z 2022-11-23T02:38:36.2338312Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2339132Z Ran 1 test in 0.577s 2022-11-23T02:38:36.2339521Z 2022-11-23T02:38:36.2339724Z OK 2022-11-23T02:38:36.2340033Z 2022-11-23T02:38:36.2340318Z Generating XML reports... 2022-11-23T02:38:36.2341792Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123023620.xml 2022-11-23T02:38:36.2343411Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2345067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2346230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2347861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2349139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2349758Z 2022-11-23T02:38:36.2350018Z Running tests... 2022-11-23T02:38:36.2351152Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2352167Z test_set_get (__main__.FileStoreTest) ... ok (0.585s) 2022-11-23T02:38:36.2352698Z 2022-11-23T02:38:36.2353456Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2354314Z Ran 1 test in 0.585s 2022-11-23T02:38:36.2354707Z 2022-11-23T02:38:36.2354918Z OK 2022-11-23T02:38:36.2355245Z 2022-11-23T02:38:36.2355544Z Generating XML reports... 2022-11-23T02:38:36.2357088Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123023624.xml 2022-11-23T02:38:36.2358776Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2360512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2361739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2363380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2364640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2365248Z 2022-11-23T02:38:36.2365502Z Running tests... 2022-11-23T02:38:36.2366638Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2367686Z test_compare_set (__main__.HashStoreTest) ... ok (0.585s) 2022-11-23T02:38:36.2368243Z 2022-11-23T02:38:36.2368994Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2369854Z Ran 1 test in 0.586s 2022-11-23T02:38:36.2370262Z 2022-11-23T02:38:36.2370477Z OK 2022-11-23T02:38:36.2371284Z 2022-11-23T02:38:36.2371712Z Generating XML reports... 2022-11-23T02:38:36.2373328Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123023628.xml 2022-11-23T02:38:36.2375243Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2377011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2378249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2379889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2381174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2381780Z 2022-11-23T02:38:36.2382021Z Running tests... 2022-11-23T02:38:36.2383168Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2384183Z test_set_get (__main__.HashStoreTest) ... ok (0.634s) 2022-11-23T02:38:36.2384710Z 2022-11-23T02:38:36.2385459Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2386282Z Ran 1 test in 0.634s 2022-11-23T02:38:36.2386467Z 2022-11-23T02:38:36.2386573Z OK 2022-11-23T02:38:36.2386724Z 2022-11-23T02:38:36.2386834Z Generating XML reports... 2022-11-23T02:38:36.2387399Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123023633.xml 2022-11-23T02:38:36.2388003Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2388630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2389074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2389663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2390129Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2390349Z 2022-11-23T02:38:36.2390444Z Running tests... 2022-11-23T02:38:36.2390855Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2391245Z test_compare_set (__main__.PrefixFileStoreTest) ... ok (0.582s) 2022-11-23T02:38:36.2391463Z 2022-11-23T02:38:36.2391734Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2392053Z Ran 1 test in 0.582s 2022-11-23T02:38:36.2392203Z 2022-11-23T02:38:36.2392284Z OK 2022-11-23T02:38:36.2392405Z 2022-11-23T02:38:36.2392516Z Generating XML reports... 2022-11-23T02:38:36.2393080Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123023637.xml 2022-11-23T02:38:36.2393702Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2394329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2394771Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2395370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2395830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2396047Z 2022-11-23T02:38:36.2396143Z Running tests... 2022-11-23T02:38:36.2396540Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2396930Z test_set_get (__main__.PrefixFileStoreTest) ... ok (0.580s) 2022-11-23T02:38:36.2397135Z 2022-11-23T02:38:36.2397402Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2397717Z Ran 1 test in 0.580s 2022-11-23T02:38:36.2397867Z 2022-11-23T02:38:36.2397947Z OK 2022-11-23T02:38:36.2398068Z 2022-11-23T02:38:36.2398182Z Generating XML reports... 2022-11-23T02:38:36.2398760Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123023642.xml 2022-11-23T02:38:36.2399369Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2400067Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2400510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2401105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2401570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2401788Z 2022-11-23T02:38:36.2401887Z Running tests... 2022-11-23T02:38:36.2402304Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2402689Z test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.003s) 2022-11-23T02:38:36.2402909Z 2022-11-23T02:38:36.2403177Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2403495Z Ran 1 test in 0.003s 2022-11-23T02:38:36.2403645Z 2022-11-23T02:38:36.2403781Z OK 2022-11-23T02:38:36.2403904Z 2022-11-23T02:38:36.2404020Z Generating XML reports... 2022-11-23T02:38:36.2404591Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221123023647.xml 2022-11-23T02:38:36.2405200Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2405813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2406258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2406849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2407314Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2407531Z 2022-11-23T02:38:36.2407625Z Running tests... 2022-11-23T02:38:36.2408041Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2408441Z test_compare_set (__main__.PrefixTCPStoreTest) ... ok (0.594s) 2022-11-23T02:38:36.2408656Z 2022-11-23T02:38:36.2408913Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2409232Z Ran 1 test in 0.594s 2022-11-23T02:38:36.2409382Z 2022-11-23T02:38:36.2409461Z OK 2022-11-23T02:38:36.2409581Z 2022-11-23T02:38:36.2409692Z Generating XML reports... 2022-11-23T02:38:36.2410271Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123023651.xml 2022-11-23T02:38:36.2410966Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2411593Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2412345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2413046Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2413612Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2413876Z 2022-11-23T02:38:36.2413993Z Running tests... 2022-11-23T02:38:36.2414483Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2414946Z test_set_get (__main__.PrefixTCPStoreTest) ... ok (0.589s) 2022-11-23T02:38:36.2415188Z 2022-11-23T02:38:36.2415513Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2415873Z Ran 1 test in 0.589s 2022-11-23T02:38:36.2416054Z 2022-11-23T02:38:36.2416149Z OK 2022-11-23T02:38:36.2416295Z 2022-11-23T02:38:36.2416426Z Generating XML reports... 2022-11-23T02:38:36.2417122Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123023656.xml 2022-11-23T02:38:36.2417867Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2418717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2419248Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2419947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2420506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2420779Z 2022-11-23T02:38:36.2420890Z Running tests... 2022-11-23T02:38:36.2421384Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2421834Z test_set_get (__main__.PythonStoreTest) ... ok (0.588s) 2022-11-23T02:38:36.2422074Z 2022-11-23T02:38:36.2422393Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2422768Z Ran 1 test in 0.588s 2022-11-23T02:38:36.2422949Z 2022-11-23T02:38:36.2423029Z OK 2022-11-23T02:38:36.2423176Z 2022-11-23T02:38:36.2423372Z Generating XML reports... 2022-11-23T02:38:36.2424063Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221123023700.xml 2022-11-23T02:38:36.2424804Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2425557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2426085Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2426755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2427202Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2427418Z 2022-11-23T02:38:36.2427515Z Running tests... 2022-11-23T02:38:36.2427923Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2428314Z test_nominal (__main__.RendezvousEnvTest) ... ok (0.594s) 2022-11-23T02:38:36.2428521Z 2022-11-23T02:38:36.2428793Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2429110Z Ran 1 test in 0.594s 2022-11-23T02:38:36.2429260Z 2022-11-23T02:38:36.2429338Z OK 2022-11-23T02:38:36.2429459Z 2022-11-23T02:38:36.2429558Z Generating XML reports... 2022-11-23T02:38:36.2430124Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221123023704.xml 2022-11-23T02:38:36.2430734Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2431357Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2431796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2432385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2432843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2433070Z 2022-11-23T02:38:36.2433153Z Running tests... 2022-11-23T02:38:36.2433565Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2433957Z test_common_errors (__main__.RendezvousFileTest) ... ok (0.581s) 2022-11-23T02:38:36.2434169Z 2022-11-23T02:38:36.2434434Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2434750Z Ran 1 test in 0.581s 2022-11-23T02:38:36.2434899Z 2022-11-23T02:38:36.2434978Z OK 2022-11-23T02:38:36.2435098Z 2022-11-23T02:38:36.2435210Z Generating XML reports... 2022-11-23T02:38:36.2435765Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123023709.xml 2022-11-23T02:38:36.2436379Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2437002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2437504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2438100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2438558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2438776Z 2022-11-23T02:38:36.2438870Z Running tests... 2022-11-23T02:38:36.2439265Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2439646Z test_nominal (__main__.RendezvousFileTest) ... ok (0.587s) 2022-11-23T02:38:36.2439846Z 2022-11-23T02:38:36.2440110Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2440427Z Ran 1 test in 0.588s 2022-11-23T02:38:36.2440580Z 2022-11-23T02:38:36.2440657Z OK 2022-11-23T02:38:36.2440777Z 2022-11-23T02:38:36.2440889Z Generating XML reports... 2022-11-23T02:38:36.2441460Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123023713.xml 2022-11-23T02:38:36.2442170Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2442790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2443228Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2443812Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2444273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2444490Z 2022-11-23T02:38:36.2444585Z Running tests... 2022-11-23T02:38:36.2444993Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2445378Z test_common_errors (__main__.RendezvousTCPTest) ... ok (0.579s) 2022-11-23T02:38:36.2445590Z 2022-11-23T02:38:36.2445865Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2446185Z Ran 1 test in 0.579s 2022-11-23T02:38:36.2446333Z 2022-11-23T02:38:36.2446412Z OK 2022-11-23T02:38:36.2446534Z 2022-11-23T02:38:36.2446648Z Generating XML reports... 2022-11-23T02:38:36.2447215Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123023718.xml 2022-11-23T02:38:36.2447828Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2448438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2448881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2449466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2449933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2450157Z 2022-11-23T02:38:36.2450252Z Running tests... 2022-11-23T02:38:36.2450756Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2451544Z test_dns_timeout (__main__.RendezvousTCPTest) ... [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -3 - Temporary failure in name resolution). 2022-11-23T02:38:36.2452140Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456). 2022-11-23T02:38:36.2452492Z ok (0.598s) 2022-11-23T02:38:36.2452627Z 2022-11-23T02:38:36.2452894Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2453208Z Ran 1 test in 0.598s 2022-11-23T02:38:36.2453356Z 2022-11-23T02:38:36.2453436Z OK 2022-11-23T02:38:36.2453555Z 2022-11-23T02:38:36.2453667Z Generating XML reports... 2022-11-23T02:38:36.2454241Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123023723.xml 2022-11-23T02:38:36.2454915Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2455548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2455994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2456579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2457042Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2457262Z 2022-11-23T02:38:36.2457361Z Running tests... 2022-11-23T02:38:36.2457774Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2458147Z test_nominal (__main__.RendezvousTCPTest) ... ok (0.585s) 2022-11-23T02:38:36.2458350Z 2022-11-23T02:38:36.2458619Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2458996Z Ran 1 test in 0.585s 2022-11-23T02:38:36.2459150Z 2022-11-23T02:38:36.2459232Z OK 2022-11-23T02:38:36.2459353Z 2022-11-23T02:38:36.2459470Z Generating XML reports... 2022-11-23T02:38:36.2460042Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123023728.xml 2022-11-23T02:38:36.2460652Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2461261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2461701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2462288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2462751Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2462973Z 2022-11-23T02:38:36.2463067Z Running tests... 2022-11-23T02:38:36.2463487Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2463891Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... ok (10.784s) 2022-11-23T02:38:36.2464114Z 2022-11-23T02:38:36.2464369Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2464687Z Ran 1 test in 10.785s 2022-11-23T02:38:36.2464840Z 2022-11-23T02:38:36.2464918Z OK 2022-11-23T02:38:36.2465040Z 2022-11-23T02:38:36.2465150Z Generating XML reports... 2022-11-23T02:38:36.2465718Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123023732.xml 2022-11-23T02:38:36.2466337Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2466959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2467388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2467987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2468450Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2468670Z 2022-11-23T02:38:36.2468766Z Running tests... 2022-11-23T02:38:36.2469176Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2469563Z test_unknown_handler (__main__.RendezvousTest) ... ok (0.595s) 2022-11-23T02:38:36.2469774Z 2022-11-23T02:38:36.2470041Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2470342Z Ran 1 test in 0.595s 2022-11-23T02:38:36.2470492Z 2022-11-23T02:38:36.2470573Z OK 2022-11-23T02:38:36.2470692Z 2022-11-23T02:38:36.2470812Z Generating XML reports... 2022-11-23T02:38:36.2471372Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123023747.xml 2022-11-23T02:38:36.2472041Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2472678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2473121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2473698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2474163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2474384Z 2022-11-23T02:38:36.2474485Z Running tests... 2022-11-23T02:38:36.2474896Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2475289Z test_url_with_node_params (__main__.RendezvousTest) ... ok (0.591s) 2022-11-23T02:38:36.2475500Z 2022-11-23T02:38:36.2475765Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2476082Z Ran 1 test in 0.591s 2022-11-23T02:38:36.2476289Z 2022-11-23T02:38:36.2476357Z OK 2022-11-23T02:38:36.2476482Z 2022-11-23T02:38:36.2476599Z Generating XML reports... 2022-11-23T02:38:36.2477159Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123023751.xml 2022-11-23T02:38:36.2477767Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2478391Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2478835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2479419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2479869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2480088Z 2022-11-23T02:38:36.2480184Z Running tests... 2022-11-23T02:38:36.2480592Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2481323Z test_address_already_in_use (__main__.TCPStoreTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:33495 (errno: 98 - Address already in use). 2022-11-23T02:38:36.2482001Z [W socket.cpp:426] [c10d] The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use). 2022-11-23T02:38:36.2482466Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address. 2022-11-23T02:38:36.2482794Z ok (0.604s) 2022-11-23T02:38:36.2482931Z 2022-11-23T02:38:36.2483199Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2483503Z Ran 1 test in 0.605s 2022-11-23T02:38:36.2483650Z 2022-11-23T02:38:36.2483730Z OK 2022-11-23T02:38:36.2483850Z 2022-11-23T02:38:36.2483962Z Generating XML reports... 2022-11-23T02:38:36.2484513Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023756.xml 2022-11-23T02:38:36.2485121Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2485743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2486185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2486762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2487223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2487442Z 2022-11-23T02:38:36.2487537Z Running tests... 2022-11-23T02:38:36.2487944Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2488323Z test_compare_set (__main__.TCPStoreTest) ... ok (0.592s) 2022-11-23T02:38:36.2488524Z 2022-11-23T02:38:36.2488789Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2489090Z Ran 1 test in 0.592s 2022-11-23T02:38:36.2489241Z 2022-11-23T02:38:36.2489320Z OK 2022-11-23T02:38:36.2489497Z 2022-11-23T02:38:36.2489611Z Generating XML reports... 2022-11-23T02:38:36.2490170Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023801.xml 2022-11-23T02:38:36.2490913Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2491537Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2491982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2492558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2493020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2493239Z 2022-11-23T02:38:36.2493334Z Running tests... 2022-11-23T02:38:36.2493745Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2494351Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:38:36.2495103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T02:38:36.2495678Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:38:36.2495995Z ok (0.648s) 2022-11-23T02:38:36.2496121Z 2022-11-23T02:38:36.2496389Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2496701Z Ran 1 test in 0.648s 2022-11-23T02:38:36.2496851Z 2022-11-23T02:38:36.2496929Z OK 2022-11-23T02:38:36.2497050Z 2022-11-23T02:38:36.2497161Z Generating XML reports... 2022-11-23T02:38:36.2497713Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023806.xml 2022-11-23T02:38:36.2498321Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2498948Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2499378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2499962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2500427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2500646Z 2022-11-23T02:38:36.2500740Z Running tests... 2022-11-23T02:38:36.2501148Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2501560Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... ok (0.597s) 2022-11-23T02:38:36.2501793Z 2022-11-23T02:38:36.2502060Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2502368Z Ran 1 test in 0.597s 2022-11-23T02:38:36.2502518Z 2022-11-23T02:38:36.2502601Z OK 2022-11-23T02:38:36.2502721Z 2022-11-23T02:38:36.2502833Z Generating XML reports... 2022-11-23T02:38:36.2503392Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023810.xml 2022-11-23T02:38:36.2503992Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2504617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2505057Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2505633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2506096Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2506320Z 2022-11-23T02:38:36.2506416Z Running tests... 2022-11-23T02:38:36.2506823Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2507304Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... ok (0.606s) 2022-11-23T02:38:36.2507545Z 2022-11-23T02:38:36.2507822Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2508138Z Ran 1 test in 0.606s 2022-11-23T02:38:36.2508277Z 2022-11-23T02:38:36.2508356Z OK 2022-11-23T02:38:36.2508476Z 2022-11-23T02:38:36.2508587Z Generating XML reports... 2022-11-23T02:38:36.2509136Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023815.xml 2022-11-23T02:38:36.2509742Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2510365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2510807Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2511399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2511904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2512124Z 2022-11-23T02:38:36.2512220Z Running tests... 2022-11-23T02:38:36.2512637Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2513018Z test_multitenancy (__main__.TCPStoreTest) ... ok (0.604s) 2022-11-23T02:38:36.2513223Z 2022-11-23T02:38:36.2513488Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2513803Z Ran 1 test in 0.604s 2022-11-23T02:38:36.2513951Z 2022-11-23T02:38:36.2514019Z OK 2022-11-23T02:38:36.2514140Z 2022-11-23T02:38:36.2514252Z Generating XML reports... 2022-11-23T02:38:36.2514802Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023819.xml 2022-11-23T02:38:36.2515396Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2516030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2516470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2517059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2517514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2517735Z 2022-11-23T02:38:36.2517830Z Running tests... 2022-11-23T02:38:36.2518242Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2518631Z test_numkeys_delkeys (__main__.TCPStoreTest) ... ok (2.626s) 2022-11-23T02:38:36.2518836Z 2022-11-23T02:38:36.2519104Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2519418Z Ran 1 test in 2.626s 2022-11-23T02:38:36.2519567Z 2022-11-23T02:38:36.2519652Z OK 2022-11-23T02:38:36.2519777Z 2022-11-23T02:38:36.2519878Z Generating XML reports... 2022-11-23T02:38:36.2520428Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023824.xml 2022-11-23T02:38:36.2521027Z Test results will be stored in test-reports/python-unittest/distributed.test_store 2022-11-23T02:38:36.2521651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:38:36.2522094Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:38:36.2522681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:38:36.2523145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:38:36.2523365Z 2022-11-23T02:38:36.2523448Z Running tests... 2022-11-23T02:38:36.2523860Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2524232Z test_set_get (__main__.TCPStoreTest) ... ok (0.592s) 2022-11-23T02:38:36.2524494Z 2022-11-23T02:38:36.2524767Z ---------------------------------------------------------------------- 2022-11-23T02:38:36.2525089Z Ran 1 test in 0.593s 2022-11-23T02:38:36.2525239Z 2022-11-23T02:38:36.2525317Z OK 2022-11-23T02:38:36.2525439Z 2022-11-23T02:38:36.2525551Z Generating XML reports... 2022-11-23T02:38:36.2526088Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123023831.xml 2022-11-23T02:38:36.2526391Z 2022-11-23T02:38:36.2526799Z ##[endgroup] 2022-11-23T02:38:36.2527355Z FINISHED PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_store_p4nhgkfz) 2022-11-23T02:38:36.2527664Z 2022-11-23T02:38:36.2527906Z Running distributed/test_nccl ... [2022-11-23 02:38:36.224254] 2022-11-23T02:38:36.2528573Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:38:36.224894] 2022-11-23T02:38:42.2290313Z 2022-11-23T02:38:42.2291586Z Expand the folded group to see the log file of distributed/test_nccl 2022-11-23T02:38:42.2293696Z ##[group]PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_nccl_xl73djrs) 2022-11-23T02:38:42.2295623Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl 2022-11-23T02:38:42.2296320Z 2022-11-23T02:38:42.2296582Z Running tests... 2022-11-23T02:38:42.2297750Z ---------------------------------------------------------------------- 2022-11-23T02:38:42.2299477Z test_all_gather_cuda_bfloat16 (__main__.TestNCCLCUDA) ... libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:38:42.2300500Z ok (0.500s) 2022-11-23T02:38:42.2301371Z test_all_gather_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-11-23T02:38:42.2302455Z test_all_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-11-23T02:38:42.2303529Z test_all_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-11-23T02:38:42.2304554Z test_broadcast_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s) 2022-11-23T02:38:42.2305571Z test_broadcast_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s) 2022-11-23T02:38:42.2306597Z test_collective_errors_cuda (__main__.TestNCCLCUDA) ... ok (0.003s) 2022-11-23T02:38:42.2307604Z test_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s) 2022-11-23T02:38:42.2308581Z test_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s) 2022-11-23T02:38:42.2309605Z test_reduce_scatter_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s) 2022-11-23T02:38:42.2310638Z test_reduce_scatter_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-11-23T02:38:42.2311635Z test_unique_id_cuda (__main__.TestNCCLCUDA) ... ok (0.002s) 2022-11-23T02:38:42.2312163Z 2022-11-23T02:38:42.2312903Z ---------------------------------------------------------------------- 2022-11-23T02:38:42.2313771Z Ran 12 tests in 0.545s 2022-11-23T02:38:42.2314177Z 2022-11-23T02:38:42.2314440Z OK 2022-11-23T02:38:42.2314769Z 2022-11-23T02:38:42.2315066Z Generating XML reports... 2022-11-23T02:38:42.2316569Z Generated XML report: test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20221123023838.xml 2022-11-23T02:38:42.2317399Z 2022-11-23T02:38:42.2318131Z ##[endgroup] 2022-11-23T02:38:42.2319641Z FINISHED PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_nccl_xl73djrs) 2022-11-23T02:38:42.2320477Z 2022-11-23T02:38:42.2321159Z Running distributed/test_launcher ... [2022-11-23 02:38:42.229409] 2022-11-23T02:38:42.2323012Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:38:42.230010] 2022-11-23T02:38:47.0829689Z 2022-11-23T02:38:47.0830547Z Expand the folded group to see the log file of distributed/test_launcher 2022-11-23T02:38:47.0833182Z ##[group]PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_launcher_rus8fzmc) 2022-11-23T02:38:47.0835228Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher 2022-11-23T02:38:47.0835912Z 2022-11-23T02:38:47.0836179Z Running tests... 2022-11-23T02:38:47.0837496Z ---------------------------------------------------------------------- 2022-11-23T02:38:47.0840559Z test_launch_user_script (__main__.TestDistributedLaunch) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79488 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.575s) 2022-11-23T02:38:47.0842251Z 2022-11-23T02:38:47.0842977Z ---------------------------------------------------------------------- 2022-11-23T02:38:47.0843800Z Ran 1 test in 0.575s 2022-11-23T02:38:47.0844464Z 2022-11-23T02:38:47.0844713Z OK (skipped=1) 2022-11-23T02:38:47.0845088Z 2022-11-23T02:38:47.0845358Z Generating XML reports... 2022-11-23T02:38:47.0846937Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20221123023844.xml 2022-11-23T02:38:47.0847811Z 2022-11-23T02:38:47.0848534Z ##[endgroup] 2022-11-23T02:38:47.0850038Z FINISHED PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_launcher_rus8fzmc) 2022-11-23T02:38:47.0851224Z 2022-11-23T02:38:47.0851957Z Running distributed/test_data_parallel ... [2022-11-23 02:38:47.083303] 2022-11-23T02:38:47.0853781Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:38:47.083884] 2022-11-23T02:39:00.6274439Z 2022-11-23T02:39:00.6275711Z Expand the folded group to see the log file of distributed/test_data_parallel 2022-11-23T02:39:00.6278059Z ##[group]PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_data_parallel_r02fbv_z) 2022-11-23T02:39:00.6280581Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel 2022-11-23T02:39:00.6281417Z 2022-11-23T02:39:00.6281676Z Running tests... 2022-11-23T02:39:00.6282846Z ---------------------------------------------------------------------- 2022-11-23T02:39:00.6284532Z test_autocast (__main__.TestDataParallel) ... libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:39:00.6285563Z ok (3.357s) 2022-11-23T02:39:00.6286350Z test_data_parallel (__main__.TestDataParallel) ... ok (0.128s) 2022-11-23T02:39:00.6287498Z test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... ok (0.105s) 2022-11-23T02:39:00.6288656Z test_data_parallel_complex (__main__.TestDataParallel) ... ok (0.018s) 2022-11-23T02:39:00.6289767Z test_data_parallel_device_args (__main__.TestDataParallel) ... ok (0.006s) 2022-11-23T02:39:00.6291393Z test_data_parallel_function_deletion (__main__.TestDataParallel) ... ok (0.014s) 2022-11-23T02:39:00.6294042Z test_data_parallel_lazy_linear (__main__.TestDataParallel) ... /opt/conda/lib/python3.8/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-11-23T02:39:00.6296102Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-11-23T02:39:00.6296948Z ok (0.004s) 2022-11-23T02:39:00.6297767Z test_data_parallel_model_device (__main__.TestDataParallel) 2022-11-23T02:39:00.6298707Z Test device[0] check at forward time. ... ok (0.155s) 2022-11-23T02:39:00.6299718Z test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... ok (0.192s) 2022-11-23T02:39:00.6300871Z test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... ok (0.005s) 2022-11-23T02:39:00.6303831Z test_data_parallel_multiple_input (__main__.TestDataParallel) ... /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/comm.py:231: UserWarning: Using -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu". 2022-11-23T02:39:00.6305362Z warnings.warn( 2022-11-23T02:39:00.6305986Z ok (0.038s) 2022-11-23T02:39:00.6306814Z test_data_parallel_nested_input (__main__.TestDataParallel) ... ok (0.012s) 2022-11-23T02:39:00.6307942Z test_data_parallel_nested_output (__main__.TestDataParallel) ... ok (0.021s) 2022-11-23T02:39:00.6309043Z test_data_parallel_no_grad (__main__.TestDataParallel) ... ok (0.008s) 2022-11-23T02:39:00.6310098Z test_data_parallel_rnn (__main__.TestDataParallel) ... ok (0.217s) 2022-11-23T02:39:00.6311164Z test_data_parallel_small_back (__main__.TestDataParallel) ... ok (0.004s) 2022-11-23T02:39:00.6312317Z test_data_parallel_sparse (__main__.TestDataParallel) ... ok (0.269s) 2022-11-23T02:39:00.6314803Z test_gather_cpu (__main__.TestDataParallel) ... /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector. 2022-11-23T02:39:00.6316949Z warnings.warn('Was asked to gather along dimension 0, but all ' 2022-11-23T02:39:00.6317756Z ok (0.081s) 2022-11-23T02:39:00.6318590Z test_gather_different_len_dicts (__main__.TestDataParallel) ... ok (0.001s) 2022-11-23T02:39:00.6319634Z test_gather_gpu (__main__.TestDataParallel) ... ok (0.056s) 2022-11-23T02:39:00.6320636Z test_parallel_apply (__main__.TestDataParallel) ... ok (0.008s) 2022-11-23T02:39:00.6321691Z test_parallel_apply_autocast (__main__.TestDataParallel) ... ok (0.007s) 2022-11-23T02:39:00.6322801Z test_parallel_apply_passes_exception (__main__.TestDataParallel) ... ok (0.004s) 2022-11-23T02:39:00.6323934Z test_parameter_list_dict_replica (__main__.TestDataParallel) ... ok (0.013s) 2022-11-23T02:39:00.6324973Z test_replicate (__main__.TestDataParallel) ... ok (0.007s) 2022-11-23T02:39:00.6325998Z test_replicate_buffers (__main__.TestDataParallel) ... ok (0.005s) 2022-11-23T02:39:00.6327057Z test_save_replica_module (__main__.TestDataParallel) ... ok (0.003s) 2022-11-23T02:39:00.6328075Z test_scatter_cpu (__main__.TestDataParallel) ... ok (0.035s) 2022-11-23T02:39:00.6329054Z test_scatter_gpu (__main__.TestDataParallel) ... ok (0.027s) 2022-11-23T02:39:00.6330497Z test_strided_grad_layout (__main__.TestDataParallel) ... [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T02:39:00.6331813Z ok (2.093s) 2022-11-23T02:39:00.6332577Z test_zero_grad (__main__.TestDataParallel) ... ok (0.017s) 2022-11-23T02:39:00.6333729Z test_data_parallel_module_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.014s) 2022-11-23T02:39:00.6335083Z test_data_parallel_module_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.009s) 2022-11-23T02:39:00.6336440Z test_data_parallel_module_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.006s) 2022-11-23T02:39:00.6337843Z test_data_parallel_module_kwargs_only_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.072s) 2022-11-23T02:39:00.6339275Z test_data_parallel_module_kwargs_only_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.070s) 2022-11-23T02:39:00.6340704Z test_data_parallel_module_kwargs_only_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.072s) 2022-11-23T02:39:00.6342174Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.066s) 2022-11-23T02:39:00.6343685Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.066s) 2022-11-23T02:39:00.6345187Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.066s) 2022-11-23T02:39:00.6346838Z test_data_parallel_module_kwargs_only_empty_list_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.066s) 2022-11-23T02:39:00.6348363Z test_data_parallel_module_kwargs_only_empty_list_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.066s) 2022-11-23T02:39:00.6349868Z test_data_parallel_module_kwargs_only_empty_list_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.067s) 2022-11-23T02:39:00.6351370Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.067s) 2022-11-23T02:39:00.6352871Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.091s) 2022-11-23T02:39:00.6354387Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.072s) 2022-11-23T02:39:00.6355181Z 2022-11-23T02:39:00.6355971Z ---------------------------------------------------------------------- 2022-11-23T02:39:00.6356985Z Ran 46 tests in 7.792s 2022-11-23T02:39:00.6357400Z 2022-11-23T02:39:00.6357615Z OK 2022-11-23T02:39:00.6357948Z 2022-11-23T02:39:00.6358246Z Generating XML reports... 2022-11-23T02:39:00.6359881Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20221123023849.xml 2022-11-23T02:39:00.6362112Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20221123023849.xml 2022-11-23T02:39:00.6363144Z 2022-11-23T02:39:00.6363927Z ##[endgroup] 2022-11-23T02:39:00.6365545Z FINISHED PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_data_parallel_r02fbv_z) 2022-11-23T02:39:00.6366473Z 2022-11-23T02:39:00.6367205Z Running distributed/test_c10d_spawn_nccl ... [2022-11-23 02:39:00.627542] 2022-11-23T02:39:00.6369204Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:39:00.628187] 2022-11-23T02:41:18.7348151Z 2022-11-23T02:41:18.7349420Z Expand the folded group to see the log file of distributed/test_c10d_spawn_nccl 2022-11-23T02:41:18.7351826Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_nccl_glt_2nvh) 2022-11-23T02:41:18.7357165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynvixj4o 2022-11-23T02:41:18.7358640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynvixj4o/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7361016Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_reduce_nccl>]> 2022-11-23T02:41:18.7364086Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:41:18.7365220Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:41:18.7366272Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:41:18.7367329Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-11-23T02:41:18.7368274Z 2022-11-23T02:41:18.7369071Z 2022-11-23T02:41:18.7373955Z , <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]> 2022-11-23T02:41:18.7377390Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:41:18.7378414Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:41:18.7379474Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:41:18.7380806Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:41:18.7382249Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:41:18.7383497Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:41:18.7384805Z test_reduce (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:41:18.7386190Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:41:18.7387688Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) 2022-11-23T02:41:18.7390676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7392123Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7394243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7396511Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7398105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3k3cu16 2022-11-23T02:41:18.7399722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3k3cu16/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7401684Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7402650Z 2022-11-23T02:41:18.7402893Z Running tests... 2022-11-23T02:41:18.7404006Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7406065Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7407490Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7409048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7410247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7411530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgf8hati 2022-11-23T02:41:18.7412896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgf8hati/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7414636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7415796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7417369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7418576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7419734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvbfqr7i 2022-11-23T02:41:18.7421092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvbfqr7i/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7422562Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7423874Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7424685Z ok (5.360s) 2022-11-23T02:41:18.7425037Z 2022-11-23T02:41:18.7425745Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7426561Z Ran 1 test in 5.361s 2022-11-23T02:41:18.7426934Z 2022-11-23T02:41:18.7427143Z OK 2022-11-23T02:41:18.7427450Z 2022-11-23T02:41:18.7427941Z Generating XML reports... 2022-11-23T02:41:18.7429650Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123023905.xml 2022-11-23T02:41:18.7431304Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:41:18.7433122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7434276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7435822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7437013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7438185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjt9ihcw2 2022-11-23T02:41:18.7439701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjt9ihcw2/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7441312Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7441995Z 2022-11-23T02:41:18.7442234Z Running tests... 2022-11-23T02:41:18.7443303Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7445302Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7446724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7448254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7449465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7450764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhhzvh_j 2022-11-23T02:41:18.7452139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhhzvh_j/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7453879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7455029Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7456568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7457762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7458933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcov9rpii 2022-11-23T02:41:18.7460297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcov9rpii/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7461862Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7463264Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7464112Z ok (6.169s) 2022-11-23T02:41:18.7464482Z 2022-11-23T02:41:18.7465222Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7466061Z Ran 1 test in 6.169s 2022-11-23T02:41:18.7466468Z 2022-11-23T02:41:18.7466676Z OK 2022-11-23T02:41:18.7467001Z 2022-11-23T02:41:18.7467304Z Generating XML reports... 2022-11-23T02:41:18.7469047Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123023915.xml 2022-11-23T02:41:18.7470809Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:41:18.7472708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7473925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7475711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7476993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7478230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5_t215m 2022-11-23T02:41:18.7479649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5_t215m/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7481327Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7482060Z 2022-11-23T02:41:18.7482310Z Running tests... 2022-11-23T02:41:18.7483436Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7485542Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7487193Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7488838Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7490111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7491639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl45xpznx 2022-11-23T02:41:18.7493070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl45xpznx/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7494918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7496138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7497754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7499044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7500283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sgga5mu 2022-11-23T02:41:18.7501206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sgga5mu/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7501812Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7502309Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7502614Z ok (5.575s) 2022-11-23T02:41:18.7502739Z 2022-11-23T02:41:18.7503002Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7503312Z Ran 1 test in 5.576s 2022-11-23T02:41:18.7503458Z 2022-11-23T02:41:18.7503535Z OK 2022-11-23T02:41:18.7503653Z 2022-11-23T02:41:18.7503761Z Generating XML reports... 2022-11-23T02:41:18.7504387Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123023926.xml 2022-11-23T02:41:18.7505023Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:41:18.7505700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7506125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7506699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7507152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7507592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54xfaysi 2022-11-23T02:41:18.7508100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54xfaysi/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7508699Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7509029Z 2022-11-23T02:41:18.7509124Z Running tests... 2022-11-23T02:41:18.7509524Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7510274Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) ... /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7510802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7511377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7511829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7512268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1_2d93y 2022-11-23T02:41:18.7512770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1_2d93y/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7513476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7513900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7514479Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7514934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7515375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrf5tgku 2022-11-23T02:41:18.7515887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrf5tgku/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7516441Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7516939Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7517239Z ok (5.282s) 2022-11-23T02:41:18.7517375Z 2022-11-23T02:41:18.7517641Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7517954Z Ran 1 test in 5.283s 2022-11-23T02:41:18.7518101Z 2022-11-23T02:41:18.7518177Z OK 2022-11-23T02:41:18.7518294Z 2022-11-23T02:41:18.7518402Z Generating XML reports... 2022-11-23T02:41:18.7519027Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123023937.xml 2022-11-23T02:41:18.7519653Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] 2022-11-23T02:41:18.7520321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7520755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7521336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7521799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7522242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmf5vkm8 2022-11-23T02:41:18.7522755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmf5vkm8/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7523355Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7523615Z 2022-11-23T02:41:18.7523699Z Running tests... 2022-11-23T02:41:18.7524101Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7524619Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71855 2022-11-23T02:41:18.7525151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71856 2022-11-23T02:41:18.7525765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7526267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7526854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7527310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7527745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqv_9pbh 2022-11-23T02:41:18.7528261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqv_9pbh/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7528910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7529345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7529931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7530438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7530930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplwrqkx3y 2022-11-23T02:41:18.7531479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplwrqkx3y/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7532059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:41:18.7532619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:41:18.7533184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:41:18.7533749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:41:18.7534547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7535392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7536074Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7536660Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7537027Z ok (5.039s) 2022-11-23T02:41:18.7537190Z 2022-11-23T02:41:18.7537508Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7537877Z Ran 1 test in 5.040s 2022-11-23T02:41:18.7538058Z 2022-11-23T02:41:18.7538146Z OK 2022-11-23T02:41:18.7538290Z 2022-11-23T02:41:18.7538420Z Generating XML reports... 2022-11-23T02:41:18.7539178Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123023948.xml 2022-11-23T02:41:18.7540059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7540591Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7541258Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7541712Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7542155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcp80_trz 2022-11-23T02:41:18.7542669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcp80_trz/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7543262Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7543522Z 2022-11-23T02:41:18.7543615Z Running tests... 2022-11-23T02:41:18.7544016Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7544540Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72073 2022-11-23T02:41:18.7545144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72074 2022-11-23T02:41:18.7545766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7546201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7546781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7547230Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7547668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_ysux_8 2022-11-23T02:41:18.7548177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_ysux_8/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7548822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7549314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7549895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7550349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7550778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_89wnm1y 2022-11-23T02:41:18.7551280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_89wnm1y/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7551771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:41:18.7552239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:41:18.7552704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:41:18.7553175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:41:18.7553838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7554532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7555086Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7555579Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7556446Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:41:18.7556974Z warnings.warn( 2022-11-23T02:41:18.7557767Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2923: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead. 2022-11-23T02:41:18.7558302Z warnings.warn( 2022-11-23T02:41:18.7559076Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T02:41:18.7559603Z warnings.warn( 2022-11-23T02:41:18.7560383Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2923: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead. 2022-11-23T02:41:18.7560911Z warnings.warn( 2022-11-23T02:41:18.7561130Z ok (6.041s) 2022-11-23T02:41:18.7561263Z 2022-11-23T02:41:18.7561532Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7561843Z Ran 1 test in 6.041s 2022-11-23T02:41:18.7561989Z 2022-11-23T02:41:18.7562149Z OK 2022-11-23T02:41:18.7562268Z 2022-11-23T02:41:18.7562376Z Generating XML reports... 2022-11-23T02:41:18.7563008Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123023957.xml 2022-11-23T02:41:18.7563737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7564176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7564758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7565212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7565654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyr58oy2 2022-11-23T02:41:18.7566166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyr58oy2/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7566824Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7567076Z 2022-11-23T02:41:18.7567168Z Running tests... 2022-11-23T02:41:18.7567572Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7568086Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72291 2022-11-23T02:41:18.7568615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72292 2022-11-23T02:41:18.7569227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7569659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7570238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7570744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7571188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpukhlyqts 2022-11-23T02:41:18.7571800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpukhlyqts/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7572575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7573096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7573789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7574337Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7574862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bc3sr_x 2022-11-23T02:41:18.7575480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bc3sr_x/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7576071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:41:18.7576635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:41:18.7577193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:41:18.7577755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:41:18.7578542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7579377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7580045Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7580638Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7581088Z ok (5.142s) 2022-11-23T02:41:18.7581223Z 2022-11-23T02:41:18.7581496Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7581809Z Ran 1 test in 5.143s 2022-11-23T02:41:18.7581955Z 2022-11-23T02:41:18.7582031Z OK 2022-11-23T02:41:18.7582148Z 2022-11-23T02:41:18.7582247Z Generating XML reports... 2022-11-23T02:41:18.7582883Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123024008.xml 2022-11-23T02:41:18.7583607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7584045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7584623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7585079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7585578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpty6x9xw0 2022-11-23T02:41:18.7586092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpty6x9xw0/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7586686Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7586947Z 2022-11-23T02:41:18.7587039Z Running tests... 2022-11-23T02:41:18.7587439Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7587966Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72509 2022-11-23T02:41:18.7588496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72510 2022-11-23T02:41:18.7589109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7589549Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7590118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7590575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7591018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9dl_ycx 2022-11-23T02:41:18.7591529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9dl_ycx/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7592178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7592613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7593195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7593655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7594085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1j_h4c37 2022-11-23T02:41:18.7594593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1j_h4c37/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7595075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:41:18.7595541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:41:18.7596008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:41:18.7596475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:41:18.7597128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7597859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7598436Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7598934Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7599240Z ok (6.240s) 2022-11-23T02:41:18.7599374Z 2022-11-23T02:41:18.7599639Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7599950Z Ran 1 test in 6.240s 2022-11-23T02:41:18.7600098Z 2022-11-23T02:41:18.7600173Z OK 2022-11-23T02:41:18.7600282Z 2022-11-23T02:41:18.7600393Z Generating XML reports... 2022-11-23T02:41:18.7601022Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123024018.xml 2022-11-23T02:41:18.7601755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7602190Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7602837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7603293Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7603737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3op38wz5 2022-11-23T02:41:18.7604239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3op38wz5/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7604842Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7605101Z 2022-11-23T02:41:18.7605192Z Running tests... 2022-11-23T02:41:18.7605595Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7606115Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72727 2022-11-23T02:41:18.7606652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72728 2022-11-23T02:41:18.7607267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7607701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7608271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7608726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7609172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8sczdr2 2022-11-23T02:41:18.7609679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8sczdr2/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7610326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7610895Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7611548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7612074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7612600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar2094q2 2022-11-23T02:41:18.7613207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar2094q2/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7613791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:41:18.7614356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:41:18.7614917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:41:18.7615477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:41:18.7616340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7617183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7617860Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7618465Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7618842Z ok (5.139s) 2022-11-23T02:41:18.7618999Z 2022-11-23T02:41:18.7619318Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7619691Z Ran 1 test in 5.140s 2022-11-23T02:41:18.7619864Z 2022-11-23T02:41:18.7619952Z OK 2022-11-23T02:41:18.7620092Z 2022-11-23T02:41:18.7620225Z Generating XML reports... 2022-11-23T02:41:18.7620989Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123024028.xml 2022-11-23T02:41:18.7621818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7622252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7622831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7623285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7623720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpya_8m97n 2022-11-23T02:41:18.7624233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpya_8m97n/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7624832Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7625092Z 2022-11-23T02:41:18.7625184Z Running tests... 2022-11-23T02:41:18.7625596Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7626113Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72945 2022-11-23T02:41:18.7626642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72946 2022-11-23T02:41:18.7627256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7627682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7628265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7628719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7629162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhp9ajo5 2022-11-23T02:41:18.7629679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhp9ajo5/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7630336Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7630769Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7631338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7631790Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7632232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzj8wy8ew 2022-11-23T02:41:18.7632740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzj8wy8ew/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7633228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:41:18.7633698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:41:18.7634232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:41:18.7634700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:41:18.7635350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7636039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7636606Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7637108Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7637417Z ok (6.246s) 2022-11-23T02:41:18.7637551Z 2022-11-23T02:41:18.7637815Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7638125Z Ran 1 test in 6.247s 2022-11-23T02:41:18.7638312Z 2022-11-23T02:41:18.7638391Z OK 2022-11-23T02:41:18.7638512Z 2022-11-23T02:41:18.7638622Z Generating XML reports... 2022-11-23T02:41:18.7639262Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123024038.xml 2022-11-23T02:41:18.7639986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7640422Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7641006Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7641465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7641898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d6_n4cp 2022-11-23T02:41:18.7642407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d6_n4cp/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7643013Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7643272Z 2022-11-23T02:41:18.7643367Z Running tests... 2022-11-23T02:41:18.7643772Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7644283Z test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73163 2022-11-23T02:41:18.7644810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73164 2022-11-23T02:41:18.7645420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7645861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7646440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7646896Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7647350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohgojpw0 2022-11-23T02:41:18.7647863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohgojpw0/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7648517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7648943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7649524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7649979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7650424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5fbhxk3 2022-11-23T02:41:18.7650991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5fbhxk3/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7651609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:41:18.7652167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:41:18.7652733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:41:18.7653286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:41:18.7654077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7654905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7655584Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7656184Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7656615Z ok (6.040s) 2022-11-23T02:41:18.7656780Z 2022-11-23T02:41:18.7657102Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7657461Z Ran 1 test in 6.040s 2022-11-23T02:41:18.7657639Z 2022-11-23T02:41:18.7657728Z OK 2022-11-23T02:41:18.7657868Z 2022-11-23T02:41:18.7658002Z Generating XML reports... 2022-11-23T02:41:18.7658764Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123024049.xml 2022-11-23T02:41:18.7659636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7660161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7660858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7661395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7661846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1p4vmv__ 2022-11-23T02:41:18.7662357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1p4vmv__/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7662955Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7663214Z 2022-11-23T02:41:18.7663306Z Running tests... 2022-11-23T02:41:18.7663709Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7664229Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73381 2022-11-23T02:41:18.7664758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73382 2022-11-23T02:41:18.7665369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7665809Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7666394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7666851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7667293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplljr26j3 2022-11-23T02:41:18.7667804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplljr26j3/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7668452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7668878Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7669457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7669981Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7670489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltn4_4jf 2022-11-23T02:41:18.7671004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltn4_4jf/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7671491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:41:18.7671963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:41:18.7672431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:41:18.7672901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:41:18.7673553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7674239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7674873Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7675370Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7675678Z ok (5.040s) 2022-11-23T02:41:18.7675813Z 2022-11-23T02:41:18.7676082Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7676391Z Ran 1 test in 5.040s 2022-11-23T02:41:18.7676537Z 2022-11-23T02:41:18.7676605Z OK 2022-11-23T02:41:18.7676723Z 2022-11-23T02:41:18.7676830Z Generating XML reports... 2022-11-23T02:41:18.7677463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123024059.xml 2022-11-23T02:41:18.7678195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7678631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7679217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7679671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7680104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p2ls44s 2022-11-23T02:41:18.7680614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p2ls44s/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7681218Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-11-23T02:41:18.7681479Z 2022-11-23T02:41:18.7681575Z Running tests... 2022-11-23T02:41:18.7681977Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7682521Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73599 2022-11-23T02:41:18.7683079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73600 2022-11-23T02:41:18.7683695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7684126Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7684709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7685164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7685608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0sl3mwz 2022-11-23T02:41:18.7686117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0sl3mwz/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7686769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:41:18.7687202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:41:18.7687829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:41:18.7688286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:41:18.7688731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgg17z_pu 2022-11-23T02:41:18.7689243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgg17z_pu/_remote_module_non_scriptable.py 2022-11-23T02:41:18.7689738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T02:41:18.7690208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T02:41:18.7690783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:41:18.7691254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T02:41:18.7692015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7692705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T02:41:18.7693271Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7693769Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T02:41:18.7694076Z ok (5.039s) 2022-11-23T02:41:18.7694201Z 2022-11-23T02:41:18.7694465Z ---------------------------------------------------------------------- 2022-11-23T02:41:18.7694779Z Ran 1 test in 5.039s 2022-11-23T02:41:18.7694927Z 2022-11-23T02:41:18.7695006Z OK 2022-11-23T02:41:18.7695124Z 2022-11-23T02:41:18.7695233Z Generating XML reports... 2022-11-23T02:41:18.7695865Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123024109.xml 2022-11-23T02:41:18.7696223Z 2022-11-23T02:41:18.7698949Z ##[endgroup] 2022-11-23T02:41:18.7699606Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_spawn_nccl_glt_2nvh) 2022-11-23T02:41:18.7699996Z 2022-11-23T02:41:18.7700293Z Running distributed/test_c10d_pypg ... [2022-11-23 02:41:18.735635] 2022-11-23T02:41:18.7701106Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:41:18.736220] 2022-11-23T02:44:59.3275221Z 2022-11-23T02:44:59.3276161Z Expand the folded group to see the log file of distributed/test_c10d_pypg 2022-11-23T02:44:59.3278326Z ##[group]PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_pypg_k370or0i) 2022-11-23T02:44:59.3280624Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_pypg 2022-11-23T02:44:59.3281658Z 2022-11-23T02:44:59.3282147Z Running tests... 2022-11-23T02:44:59.3283339Z ---------------------------------------------------------------------- 2022-11-23T02:44:59.3284442Z test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3286351Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73817 2022-11-23T02:44:59.3289419Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3291070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3293147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3294636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3296014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3299104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwd_z3axf 2022-11-23T02:44:59.3327508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwd_z3axf/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3328961Z ok (6.925s) 2022-11-23T02:44:59.3329880Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3331594Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73889 2022-11-23T02:44:59.3333682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3334883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3336489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3338168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3339339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3340642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpli7u8pgs 2022-11-23T02:44:59.3342106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpli7u8pgs/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3343138Z ok (6.424s) 2022-11-23T02:44:59.3344053Z test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3345517Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73961 2022-11-23T02:44:59.3347452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3348636Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3350265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3352166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3353320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3354616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6i_2e8ql 2022-11-23T02:44:59.3356005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6i_2e8ql/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3357325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3358578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3361894Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:44:59.3363914Z warnings.warn( 2022-11-23T02:44:59.3364883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3366144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3367372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3368623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3369521Z ok (6.526s) 2022-11-23T02:44:59.3370426Z test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3372432Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74033 2022-11-23T02:44:59.3374385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3375570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3377141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3378378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3379531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3380418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8x2pc9m9 2022-11-23T02:44:59.3381028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8x2pc9m9/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3381528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3382065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3383272Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:44:59.3383993Z warnings.warn( 2022-11-23T02:44:59.3384348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3384824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3385302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3385776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3386132Z ok (6.627s) 2022-11-23T02:44:59.3386503Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3387230Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74105 2022-11-23T02:44:59.3387931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3388383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3388976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3389448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3389886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3390389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi51in9sm 2022-11-23T02:44:59.3390912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi51in9sm/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3391402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3391876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3392220Z ok (6.425s) 2022-11-23T02:44:59.3392590Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3393309Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74177 2022-11-23T02:44:59.3394027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3394471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3395116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3395592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3396030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3396525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4da5_ccz 2022-11-23T02:44:59.3397049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4da5_ccz/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3397551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3398024Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3398355Z ok (6.424s) 2022-11-23T02:44:59.3398705Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3399502Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74249 2022-11-23T02:44:59.3400228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3400681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3401275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3401745Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3402180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3402661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0uicwiy 2022-11-23T02:44:59.3403188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0uicwiy/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3403697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3404723Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:44:59.3405722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3406065Z ok (6.626s) 2022-11-23T02:44:59.3406415Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3407154Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74321 2022-11-23T02:44:59.3407881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3408329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3408920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3409377Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3409811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3410305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjkpxe8t 2022-11-23T02:44:59.3410892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjkpxe8t/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3411267Z ok (6.426s) 2022-11-23T02:44:59.3411745Z test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3412424Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74393 2022-11-23T02:44:59.3413289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3413827Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3414542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3415098Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3415622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3416205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebc3ou6w 2022-11-23T02:44:59.3416911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebc3ou6w/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3417519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3418078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3418487Z ok (6.728s) 2022-11-23T02:44:59.3418926Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3419614Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74465 2022-11-23T02:44:59.3420487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3420934Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3421530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3421989Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3422432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3422924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25gbg6rc 2022-11-23T02:44:59.3423443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25gbg6rc/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3424495Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:44:59.3426211Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:44:59.3426936Z warnings.warn( 2022-11-23T02:44:59.3427302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3427781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3428124Z ok (6.528s) 2022-11-23T02:44:59.3428473Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3429111Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74537 2022-11-23T02:44:59.3429843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3430294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3430889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3431355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3431794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3432282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mb4zqlw 2022-11-23T02:44:59.3432796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mb4zqlw/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3434025Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:44:59.3434786Z warnings.warn( 2022-11-23T02:44:59.3435154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3435630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3435980Z ok (6.427s) 2022-11-23T02:44:59.3436343Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3436891Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74609 2022-11-23T02:44:59.3437582Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3438035Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3438630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3439097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3439533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3440024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40n7f_l1 2022-11-23T02:44:59.3440545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40n7f_l1/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3441046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3441508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3441857Z ok (6.426s) 2022-11-23T02:44:59.3442226Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkSubclass) 2022-11-23T02:44:59.3442776Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74681 2022-11-23T02:44:59.3443471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3443918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3444513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3444966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3445403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3445901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0aw9507 2022-11-23T02:44:59.3446521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0aw9507/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3447023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3447499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3447971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3448432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3448778Z ok (6.427s) 2022-11-23T02:44:59.3449217Z test_ddp_invoke_work_object (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74753 2022-11-23T02:44:59.3449938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3450442Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3451099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3451636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3452135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3452714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7bknujk 2022-11-23T02:44:59.3453331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7bknujk/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3453771Z ok (4.021s) 2022-11-23T02:44:59.3454284Z test_ddp_with_pypg (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74822 2022-11-23T02:44:59.3455130Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3455681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3456379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3456942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3457474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3458059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg68pmipf 2022-11-23T02:44:59.3458688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg68pmipf/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3459302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3459712Z ok (4.421s) 2022-11-23T02:44:59.3460237Z test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74891 2022-11-23T02:44:59.3461112Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3461647Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3462354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3462923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3463453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3464043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w5edxm9 2022-11-23T02:44:59.3464669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w5edxm9/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3465254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3465671Z ok (4.020s) 2022-11-23T02:44:59.3466291Z test_invalid_powerSGD_state (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74960 2022-11-23T02:44:59.3467156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3467691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3468397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3468954Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3469463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3470397Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3471526Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3472603Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3473676Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3474762Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3475820Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3476453Z ok (4.020s) 2022-11-23T02:44:59.3476892Z test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75027 2022-11-23T02:44:59.3477618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3478066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3478659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3479127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3479561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3480048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzsgaom9 2022-11-23T02:44:59.3480561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzsgaom9/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3481243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3481588Z ok (4.423s) 2022-11-23T02:44:59.3482040Z test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75098 2022-11-23T02:44:59.3482771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3483217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3483809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3484263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3484698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3485251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf75751tv 2022-11-23T02:44:59.3485770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf75751tv/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3486272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3486615Z ok (4.324s) 2022-11-23T02:44:59.3486946Z test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3487653Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75169 2022-11-23T02:44:59.3488373Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3488823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3489422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3525349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3526617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3528075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvif07eom 2022-11-23T02:44:59.3529539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvif07eom/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3530958Z ok (6.323s) 2022-11-23T02:44:59.3531917Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3533467Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75241 2022-11-23T02:44:59.3535745Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3537019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3538707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3540003Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3541198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3542549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40bi0omh 2022-11-23T02:44:59.3543985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40bi0omh/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3544990Z ok (6.423s) 2022-11-23T02:44:59.3545919Z test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3547424Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75313 2022-11-23T02:44:59.3549923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3551167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3552828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3554109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3555298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3556641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdaloq3_r 2022-11-23T02:44:59.3558071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdaloq3_r/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3559452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3560747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3564362Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:44:59.3566382Z warnings.warn( 2022-11-23T02:44:59.3567376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3568676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3569969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3571403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3572326Z ok (6.730s) 2022-11-23T02:44:59.3573272Z test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3574764Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75385 2022-11-23T02:44:59.3576725Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3577949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3579580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3580845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3582033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3583369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxtpvbou 2022-11-23T02:44:59.3584821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxtpvbou/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3586202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3587490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3590845Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:44:59.3592820Z warnings.warn( 2022-11-23T02:44:59.3593811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3595094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3596552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3597856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3598777Z ok (6.427s) 2022-11-23T02:44:59.3599770Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3601779Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75457 2022-11-23T02:44:59.3603754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3604963Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3606596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3608015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3609211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3610644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnllx3t1 2022-11-23T02:44:59.3612076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnllx3t1/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3613452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3614722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3615642Z ok (6.424s) 2022-11-23T02:44:59.3616632Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3618646Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75529 2022-11-23T02:44:59.3620624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3621833Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3623476Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3624788Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3625983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3627337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmf8xo63 2022-11-23T02:44:59.3628769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmf8xo63/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3630134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3631428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3632363Z ok (6.224s) 2022-11-23T02:44:59.3633283Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3635307Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75601 2022-11-23T02:44:59.3637305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3638517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3640143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3641417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3642597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3644117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cn9ot7g 2022-11-23T02:44:59.3645533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cn9ot7g/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3646895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3649747Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:44:59.3652815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3653895Z ok (6.343s) 2022-11-23T02:44:59.3654830Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3656870Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75673 2022-11-23T02:44:59.3658860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3659531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3660138Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3660593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3661017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3661493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0tail8sp 2022-11-23T02:44:59.3662007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0tail8sp/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3662365Z ok (6.225s) 2022-11-23T02:44:59.3662693Z test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3663230Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75745 2022-11-23T02:44:59.3663942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3664372Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3664949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3665402Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3665826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3666300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplz_w5eni 2022-11-23T02:44:59.3666806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplz_w5eni/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3667286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3667747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3668078Z ok (6.325s) 2022-11-23T02:44:59.3668426Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3668985Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75817 2022-11-23T02:44:59.3669690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3670195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3670771Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3671224Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3671647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3672126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0o0ram2 2022-11-23T02:44:59.3672630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0o0ram2/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3673658Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T02:44:59.3675429Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:44:59.3676135Z warnings.warn( 2022-11-23T02:44:59.3676490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3676950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3677285Z ok (6.325s) 2022-11-23T02:44:59.3677623Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3678180Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75889 2022-11-23T02:44:59.3678883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3679315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3679895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3680352Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3680774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3681245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje59_nm3 2022-11-23T02:44:59.3681757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje59_nm3/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3682980Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T02:44:59.3683685Z warnings.warn( 2022-11-23T02:44:59.3684037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3684498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3684830Z ok (6.424s) 2022-11-23T02:44:59.3685180Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3685762Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75961 2022-11-23T02:44:59.3686444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3686875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3687452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3687901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3688320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3688800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1lf5bhc 2022-11-23T02:44:59.3689311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1lf5bhc/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3689857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3690318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3690704Z ok (6.227s) 2022-11-23T02:44:59.3691051Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkWrapper) 2022-11-23T02:44:59.3691583Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76033 2022-11-23T02:44:59.3692265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3692700Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3693270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3693724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3694153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3694633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzx5ytke 2022-11-23T02:44:59.3695142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzx5ytke/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3695633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3696094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3696546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3697009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3697341Z ok (6.225s) 2022-11-23T02:44:59.3697772Z test_ddp_invoke_work_object (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76105 2022-11-23T02:44:59.3698481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3698913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3699485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3699932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3700353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3700832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmzgei1a 2022-11-23T02:44:59.3701345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmzgei1a/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3701706Z ok (3.822s) 2022-11-23T02:44:59.3702186Z test_ddp_with_pypg (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76174 2022-11-23T02:44:59.3702884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3703320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3703888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3704341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3704772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3705281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pevngn9 2022-11-23T02:44:59.3705799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pevngn9/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3706289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3706735Z ok (3.819s) 2022-11-23T02:44:59.3707160Z test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76243 2022-11-23T02:44:59.3707870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3708300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3708881Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3709333Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3709756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3710236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0c6ze7v 2022-11-23T02:44:59.3710742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0c6ze7v/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3711229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3711564Z ok (3.818s) 2022-11-23T02:44:59.3711995Z test_invalid_powerSGD_state (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76312 2022-11-23T02:44:59.3712697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3713130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3713711Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3714155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3714578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3715359Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3716424Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3717537Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3718599Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3719649Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3720701Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T02:44:59.3721353Z ok (3.821s) 2022-11-23T02:44:59.3721787Z test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76379 2022-11-23T02:44:59.3722500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3722937Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3723519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3723973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3724396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3724882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgn88dfvh 2022-11-23T02:44:59.3725385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgn88dfvh/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3725878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3726211Z ok (4.222s) 2022-11-23T02:44:59.3726646Z test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76450 2022-11-23T02:44:59.3727351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T02:44:59.3727787Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T02:44:59.3728368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T02:44:59.3728814Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T02:44:59.3729239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T02:44:59.3729721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucdiahg6 2022-11-23T02:44:59.3730229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucdiahg6/_remote_module_non_scriptable.py 2022-11-23T02:44:59.3730767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T02:44:59.3731099Z ok (4.323s) 2022-11-23T02:44:59.3731233Z 2022-11-23T02:44:59.3731536Z ---------------------------------------------------------------------- 2022-11-23T02:44:59.3731846Z Ran 38 tests in 216.647s 2022-11-23T02:44:59.3731997Z 2022-11-23T02:44:59.3732072Z OK 2022-11-23T02:44:59.3732190Z 2022-11-23T02:44:59.3732300Z Generating XML reports... 2022-11-23T02:44:59.3732902Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20221123024120.xml 2022-11-23T02:44:59.3733741Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20221123024120.xml 2022-11-23T02:44:59.3734072Z 2022-11-23T02:44:59.3734497Z ##[endgroup] 2022-11-23T02:44:59.3735075Z FINISHED PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_pypg_k370or0i) 2022-11-23T02:44:59.3735390Z 2022-11-23T02:44:59.3735633Z Running distributed/test_c10d_nccl ... [2022-11-23 02:44:59.330137] 2022-11-23T02:44:59.3736335Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:44:59.331248] 2022-11-23T03:04:48.3807628Z 2022-11-23T03:04:48.3808848Z Expand the folded group to see the log file of distributed/test_c10d_nccl 2022-11-23T03:04:48.3812238Z ##[group]PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_nccl_6kia0ius) 2022-11-23T03:04:48.3821284Z , <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]> 2022-11-23T03:04:48.3827912Z test_all_reduce_coalesced_nccl (__main__.CommTest) 2022-11-23T03:04:48.3829319Z test_broadcast_coalesced_nccl (__main__.CommTest) 2022-11-23T03:04:48.3830608Z test_nccl_barrier (__main__.CommTest) 2022-11-23T03:04:48.3831714Z test_nccl_barrier_device_ids (__main__.CommTest) 2022-11-23T03:04:48.3832882Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) 2022-11-23T03:04:48.3834035Z test_nccl_barrier_timeout (__main__.CommTest) 2022-11-23T03:04:48.3835418Z test_nccl_barrier_timeout_new_group (__main__.CommTest) 2022-11-23T03:04:48.3837066Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) 2022-11-23T03:04:48.3838904Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) 2022-11-23T03:04:48.3840548Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) 2022-11-23T03:04:48.3842276Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) 2022-11-23T03:04:48.3843468Z test_nncl_rank_membership (__main__.CommTest) 2022-11-23T03:04:48.3844964Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) 2022-11-23T03:04:48.3846670Z test_sequence_num_incremented_nccl_default (__main__.CommTest) 2022-11-23T03:04:48.3848215Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) 2022-11-23T03:04:48.3849632Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) 2022-11-23T03:04:48.3851239Z test_sequence_num_set_nccl_new_group (__main__.CommTest) 2022-11-23T03:04:48.3852227Z test_tensor_dtype_complex (__main__.CommTest) 2022-11-23T03:04:48.3853641Z test_tensor_dtype_mismatch (__main__.CommTest) 2022-11-23T03:04:48.3857986Z , <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]> 2022-11-23T03:04:48.3861498Z test_allgather_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:04:48.3862941Z test_allreduce_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:04:48.3864437Z test_broadcast_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:04:48.3865909Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:04:48.3867440Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) 2022-11-23T03:04:48.3868998Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:04:48.3870381Z test_scatter_work_wait_gpu (__main__.CompilerTest) 2022-11-23T03:04:48.3903671Z , <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_packed_sequence>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]> 2022-11-23T03:04:48.3925528Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3926745Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3927978Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3929173Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3930253Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3931522Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3932591Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3933721Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3934838Z test_channels_last_contig (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3935927Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3937316Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3938558Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3939767Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3940980Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3942437Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3943864Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3945229Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3946556Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3947884Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3949664Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3951189Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3952696Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3954047Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3955357Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3956702Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3958057Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3959310Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3960623Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3961863Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3963018Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3964131Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3965137Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3966150Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3967114Z test_failure_recovery (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3968141Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3969227Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3970310Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3971796Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3972888Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3973945Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3974808Z test_fp16 (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3975659Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3976588Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3977462Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3978415Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3979392Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3980256Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3981159Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3981740Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3982341Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3982935Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3983492Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3984010Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3984534Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3985117Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3985709Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3986371Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3986935Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3987424Z test_no_grad (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3987917Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3988398Z test_pass_default_pg (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3988900Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3989447Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3989995Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3990515Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.3990976Z 2022-11-23T03:04:48.3992494Z , <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]> 2022-11-23T03:04:48.3993997Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) 2022-11-23T03:04:48.3994499Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) 2022-11-23T03:04:48.3994984Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) 2022-11-23T03:04:48.3995463Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) 2022-11-23T03:04:48.3995975Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) 2022-11-23T03:04:48.3996472Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) 2022-11-23T03:04:48.3996956Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) 2022-11-23T03:04:48.3997436Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) 2022-11-23T03:04:48.3997891Z test_nccl_timeout (__main__.NcclErrorHandlingTest) 2022-11-23T03:04:48.3999081Z , <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_reduce_scatter_base>]> 2022-11-23T03:04:48.4000392Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) 2022-11-23T03:04:48.4001017Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) 2022-11-23T03:04:48.4001610Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) 2022-11-23T03:04:48.4002235Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) 2022-11-23T03:04:48.4002873Z ]> 2022-11-23T03:04:48.4003423Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) 2022-11-23T03:04:48.4005845Z , <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_nccl_dist_backend_error>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]> 2022-11-23T03:04:48.4008454Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4008918Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4009378Z test_allgather_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4009797Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4010226Z test_barrier (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4010748Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4011189Z test_empty_tensors (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4011622Z test_gather_checks (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4012046Z test_gather_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4012460Z test_gather_stress (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4012917Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4013367Z test_reduce_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4013825Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4016895Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4017371Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4017799Z test_scatter_checks (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4018241Z test_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4018672Z test_scatter_stress (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4019103Z test_send_recv (__main__.ProcessGroupNCCLTest) 2022-11-23T03:04:48.4019611Z ]> 2022-11-23T03:04:48.4020110Z test_common_errors (__main__.RendezvousEnvTest) 2022-11-23T03:04:48.4020491Z 2022-11-23T03:04:48.4020998Z ]> 2022-11-23T03:04:48.4021515Z test_default_store_timeout_nccl (__main__.TimeoutTest) 2022-11-23T03:04:48.4022465Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4023210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4023683Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4024305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4024766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4024996Z 2022-11-23T03:04:48.4025099Z Running tests... 2022-11-23T03:04:48.4025541Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4026059Z test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76654 2022-11-23T03:04:48.4026587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76655 2022-11-23T03:04:48.4027228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4027758Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4028369Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4028827Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4029276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4029926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4030384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4030989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4031465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4031915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4032450Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4032973Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4033992Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:04:48.4034618Z warnings.warn( 2022-11-23T03:04:48.4035534Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:04:48.4036154Z warnings.warn( 2022-11-23T03:04:48.4036405Z ok (5.205s) 2022-11-23T03:04:48.4036548Z 2022-11-23T03:04:48.4036833Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4037148Z Ran 1 test in 5.206s 2022-11-23T03:04:48.4037306Z 2022-11-23T03:04:48.4037392Z OK 2022-11-23T03:04:48.4037522Z 2022-11-23T03:04:48.4037646Z Generating XML reports... 2022-11-23T03:04:48.4038211Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024502.xml 2022-11-23T03:04:48.4038837Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4039487Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4039942Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4040527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4041005Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4041297Z 2022-11-23T03:04:48.4041406Z Running tests... 2022-11-23T03:04:48.4041843Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4042359Z test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76870 2022-11-23T03:04:48.4042887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76871 2022-11-23T03:04:48.4043529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4043965Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4044572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4045048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4045560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4046213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4046672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4047272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4047748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4048175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4048724Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4049241Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4049575Z ok (5.300s) 2022-11-23T03:04:48.4049719Z 2022-11-23T03:04:48.4049996Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4050331Z Ran 1 test in 5.301s 2022-11-23T03:04:48.4050489Z 2022-11-23T03:04:48.4050648Z OK 2022-11-23T03:04:48.4050930Z 2022-11-23T03:04:48.4051068Z Generating XML reports... 2022-11-23T03:04:48.4051643Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024511.xml 2022-11-23T03:04:48.4052279Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4053060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4053605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4054331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4054886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4055167Z 2022-11-23T03:04:48.4055295Z Running tests... 2022-11-23T03:04:48.4055814Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4056409Z test_nccl_barrier (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77086 2022-11-23T03:04:48.4057021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77087 2022-11-23T03:04:48.4057792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4058346Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4059044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4059620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4060159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4061059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4061611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4062352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4062848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4063297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4063682Z skip: Need at least 4 CUDA devices (5.514s) 2022-11-23T03:04:48.4063871Z 2022-11-23T03:04:48.4064153Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4064485Z Ran 1 test in 5.515s 2022-11-23T03:04:48.4064642Z 2022-11-23T03:04:48.4064743Z OK (skipped=1) 2022-11-23T03:04:48.4064896Z 2022-11-23T03:04:48.4065013Z Generating XML reports... 2022-11-23T03:04:48.4065642Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024521.xml 2022-11-23T03:04:48.4066246Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4066894Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4067349Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4067952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4068424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4068655Z 2022-11-23T03:04:48.4068757Z Running tests... 2022-11-23T03:04:48.4069183Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4069691Z test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77287 2022-11-23T03:04:48.4070207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77288 2022-11-23T03:04:48.4070846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4071307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4071906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4072383Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4072828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4073320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4073954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4074410Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4075018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4075496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4075941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4076431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4077111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4077827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4078398Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4078978Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4079311Z ok (5.107s) 2022-11-23T03:04:48.4079454Z 2022-11-23T03:04:48.4079734Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4080060Z Ran 1 test in 5.107s 2022-11-23T03:04:48.4080218Z 2022-11-23T03:04:48.4080307Z OK 2022-11-23T03:04:48.4080435Z 2022-11-23T03:04:48.4080534Z Generating XML reports... 2022-11-23T03:04:48.4081095Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024530.xml 2022-11-23T03:04:48.4081706Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4082348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4082800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4083409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4083941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4084170Z 2022-11-23T03:04:48.4084276Z Running tests... 2022-11-23T03:04:48.4084689Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4085224Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77499 2022-11-23T03:04:48.4085769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77500 2022-11-23T03:04:48.4086403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4086861Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4087463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4087951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4088375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4088865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4089522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4089980Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4090709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4091196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4091640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4092111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4092916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4093769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4094245Z ok (4.612s) 2022-11-23T03:04:48.4094414Z 2022-11-23T03:04:48.4094742Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4095139Z Ran 1 test in 4.613s 2022-11-23T03:04:48.4095323Z 2022-11-23T03:04:48.4095428Z OK 2022-11-23T03:04:48.4095581Z 2022-11-23T03:04:48.4095705Z Generating XML reports... 2022-11-23T03:04:48.4096380Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024539.xml 2022-11-23T03:04:48.4097122Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4098061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4098621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4099349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4099921Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4100195Z 2022-11-23T03:04:48.4100299Z Running tests... 2022-11-23T03:04:48.4100809Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4101417Z test_nccl_barrier_timeout (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77704 2022-11-23T03:04:48.4102043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77705 2022-11-23T03:04:48.4102808Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4103435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4104162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4104735Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4105251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4106029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4106580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4107302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4107877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4108422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4108912Z skip: Need at least 4 CUDA devices (4.605s) 2022-11-23T03:04:48.4109114Z 2022-11-23T03:04:48.4109448Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4109850Z Ran 1 test in 4.605s 2022-11-23T03:04:48.4110038Z 2022-11-23T03:04:48.4110159Z OK (skipped=1) 2022-11-23T03:04:48.4110344Z 2022-11-23T03:04:48.4110487Z Generating XML reports... 2022-11-23T03:04:48.4111159Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024547.xml 2022-11-23T03:04:48.4111913Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4112554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4112991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4113592Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4114068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4114294Z 2022-11-23T03:04:48.4114397Z Running tests... 2022-11-23T03:04:48.4114825Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4115347Z test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77905 2022-11-23T03:04:48.4115884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77906 2022-11-23T03:04:48.4116503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4116959Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4117556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4118097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4118548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4119205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4119659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4120241Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4120715Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4121162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4121565Z skip: Need at least 4 CUDA devices (5.016s) 2022-11-23T03:04:48.4121759Z 2022-11-23T03:04:48.4122037Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4122471Z Ran 1 test in 5.017s 2022-11-23T03:04:48.4122662Z 2022-11-23T03:04:48.4122785Z OK (skipped=1) 2022-11-23T03:04:48.4122963Z 2022-11-23T03:04:48.4123086Z Generating XML reports... 2022-11-23T03:04:48.4123766Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024556.xml 2022-11-23T03:04:48.4124508Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4125285Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4125838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4126564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4127146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4127415Z 2022-11-23T03:04:48.4127549Z Running tests... 2022-11-23T03:04:48.4128052Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4129446Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/68319 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.580s) 2022-11-23T03:04:48.4130153Z 2022-11-23T03:04:48.4130485Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4130998Z Ran 1 test in 0.580s 2022-11-23T03:04:48.4131184Z 2022-11-23T03:04:48.4131305Z OK (skipped=1) 2022-11-23T03:04:48.4131487Z 2022-11-23T03:04:48.4131626Z Generating XML reports... 2022-11-23T03:04:48.4132312Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024605.xml 2022-11-23T03:04:48.4133073Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4133818Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4134366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4135089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4135659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4135931Z 2022-11-23T03:04:48.4136054Z Running tests... 2022-11-23T03:04:48.4136564Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4137187Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78172 2022-11-23T03:04:48.4137807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78173 2022-11-23T03:04:48.4138669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4139212Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4139945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4140510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4141039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4141800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4142325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4142931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4143474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4143915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4144401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4144902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4145588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4146307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4146821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:04:48.4147319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:04:48.4148003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4148712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4149300Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4149823Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4150150Z ok (5.305s) 2022-11-23T03:04:48.4150293Z 2022-11-23T03:04:48.4150552Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4150881Z Ran 1 test in 5.305s 2022-11-23T03:04:48.4151037Z 2022-11-23T03:04:48.4151124Z OK 2022-11-23T03:04:48.4151251Z 2022-11-23T03:04:48.4151370Z Generating XML reports... 2022-11-23T03:04:48.4151931Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024609.xml 2022-11-23T03:04:48.4152556Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4153197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4153631Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4154229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4154707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4154936Z 2022-11-23T03:04:48.4155037Z Running tests... 2022-11-23T03:04:48.4155467Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4155982Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78399 2022-11-23T03:04:48.4156512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78400 2022-11-23T03:04:48.4157213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4157662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4158265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4158741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4159187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4159686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4160341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4160795Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4161440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4161913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4162362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4162851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4163530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4164060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:04:48.4164734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4165269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:04:48.4165929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4166629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4167219Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4167739Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4168066Z ok (5.508s) 2022-11-23T03:04:48.4168210Z 2022-11-23T03:04:48.4168488Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4168821Z Ran 1 test in 5.508s 2022-11-23T03:04:48.4168958Z 2022-11-23T03:04:48.4169044Z OK 2022-11-23T03:04:48.4169171Z 2022-11-23T03:04:48.4169291Z Generating XML reports... 2022-11-23T03:04:48.4169853Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024619.xml 2022-11-23T03:04:48.4170485Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4171187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4171645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4172251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4172709Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4172940Z 2022-11-23T03:04:48.4173043Z Running tests... 2022-11-23T03:04:48.4173467Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4173980Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78617 2022-11-23T03:04:48.4174587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78618 2022-11-23T03:04:48.4175238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4175696Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4176280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4176755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4177201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4177694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4178345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4178801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4179469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4179923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4180365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4180856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4181530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4182067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:04:48.4182737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4183270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:04:48.4183943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4184625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4185216Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4185732Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4186061Z ok (5.102s) 2022-11-23T03:04:48.4186204Z 2022-11-23T03:04:48.4186479Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4186809Z Ran 1 test in 5.103s 2022-11-23T03:04:48.4186970Z 2022-11-23T03:04:48.4187054Z OK 2022-11-23T03:04:48.4187185Z 2022-11-23T03:04:48.4187284Z Generating XML reports... 2022-11-23T03:04:48.4187842Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024628.xml 2022-11-23T03:04:48.4188467Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4189113Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4189570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4190171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4190652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4190878Z 2022-11-23T03:04:48.4190961Z Running tests... 2022-11-23T03:04:48.4191387Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4191892Z test_nncl_rank_membership (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78835 2022-11-23T03:04:48.4192468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78836 2022-11-23T03:04:48.4193118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4193580Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4194181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4194638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4195088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4195574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4196227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4196685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4197379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4197854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4198302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4198771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4199446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4199981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:04:48.4200654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4201187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:04:48.4201866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4202569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4202967Z ok (4.703s) 2022-11-23T03:04:48.4203091Z 2022-11-23T03:04:48.4203371Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4203703Z Ran 1 test in 4.704s 2022-11-23T03:04:48.4203863Z 2022-11-23T03:04:48.4203955Z OK 2022-11-23T03:04:48.4204083Z 2022-11-23T03:04:48.4204200Z Generating XML reports... 2022-11-23T03:04:48.4204761Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024638.xml 2022-11-23T03:04:48.4205384Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4206019Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4206480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4207084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4207559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4207782Z 2022-11-23T03:04:48.4207884Z Running tests... 2022-11-23T03:04:48.4208320Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4208858Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79042 2022-11-23T03:04:48.4209396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79043 2022-11-23T03:04:48.4210012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4210666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4211292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4211771Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4212221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4212709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4213361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4213794Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4214397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4214964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4215407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4215898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4216578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4217109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:04:48.4217779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4218292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:04:48.4218966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4219676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4220263Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4220783Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4221121Z ok (5.410s) 2022-11-23T03:04:48.4221268Z 2022-11-23T03:04:48.4221545Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4221854Z Ran 1 test in 5.411s 2022-11-23T03:04:48.4222013Z 2022-11-23T03:04:48.4222099Z OK 2022-11-23T03:04:48.4222226Z 2022-11-23T03:04:48.4222343Z Generating XML reports... 2022-11-23T03:04:48.4222910Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024646.xml 2022-11-23T03:04:48.4223531Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4224180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4224630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4225211Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4225686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4225916Z 2022-11-23T03:04:48.4226019Z Running tests... 2022-11-23T03:04:48.4226445Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4226975Z test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79262 2022-11-23T03:04:48.4227521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79263 2022-11-23T03:04:48.4228222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4228667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4229279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4229756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4230198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4230686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4231343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4231800Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4232400Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4232913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4233359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4233850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4234531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4235250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4235783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:04:48.4236279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:04:48.4236928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4237646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4238226Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4238757Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4239093Z ok (5.103s) 2022-11-23T03:04:48.4239238Z 2022-11-23T03:04:48.4239516Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4239854Z Ran 1 test in 5.103s 2022-11-23T03:04:48.4240013Z 2022-11-23T03:04:48.4240099Z OK 2022-11-23T03:04:48.4240229Z 2022-11-23T03:04:48.4240328Z Generating XML reports... 2022-11-23T03:04:48.4240891Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024655.xml 2022-11-23T03:04:48.4241518Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4242178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4242636Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4243242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4243720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4243946Z 2022-11-23T03:04:48.4244029Z Running tests... 2022-11-23T03:04:48.4244457Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4244992Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79480 2022-11-23T03:04:48.4245532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79481 2022-11-23T03:04:48.4246235Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4246706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4247317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4247878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4248386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4249166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4249723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4250459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4251099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4251724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4252214Z skip: Need at least 4 CUDA devices (4.514s) 2022-11-23T03:04:48.4252439Z 2022-11-23T03:04:48.4252761Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4253169Z Ran 1 test in 4.515s 2022-11-23T03:04:48.4253365Z 2022-11-23T03:04:48.4253489Z OK (skipped=1) 2022-11-23T03:04:48.4253674Z 2022-11-23T03:04:48.4253814Z Generating XML reports... 2022-11-23T03:04:48.4254498Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024704.xml 2022-11-23T03:04:48.4255244Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4256040Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4256565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4257301Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4257876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4258149Z 2022-11-23T03:04:48.4258279Z Running tests... 2022-11-23T03:04:48.4258818Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4259446Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79681 2022-11-23T03:04:48.4260087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79682 2022-11-23T03:04:48.4260841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4261393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4262126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4262721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4263170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4263667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4264323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4264766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4265371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4265857Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4266307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4266869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4267564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4268276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4268863Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4269360Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4269696Z ok (5.125s) 2022-11-23T03:04:48.4269842Z 2022-11-23T03:04:48.4270121Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4270455Z Ran 1 test in 5.125s 2022-11-23T03:04:48.4270613Z 2022-11-23T03:04:48.4270704Z OK 2022-11-23T03:04:48.4270832Z 2022-11-23T03:04:48.4270955Z Generating XML reports... 2022-11-23T03:04:48.4271567Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024713.xml 2022-11-23T03:04:48.4272197Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4272845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4273308Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4273918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4274405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4274635Z 2022-11-23T03:04:48.4274738Z Running tests... 2022-11-23T03:04:48.4275173Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4275679Z test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79893 2022-11-23T03:04:48.4276217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79894 2022-11-23T03:04:48.4276855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4277316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4277922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4278395Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4278841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4279313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4279969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4280428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4281032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4281508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4281950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4282440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4283096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4283638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:04:48.4284314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4284910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:04:48.4285590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4286298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:04:48.4286893Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4287414Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4287722Z ok (5.203s) 2022-11-23T03:04:48.4287865Z 2022-11-23T03:04:48.4288146Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4288479Z Ran 1 test in 5.203s 2022-11-23T03:04:48.4288638Z 2022-11-23T03:04:48.4288730Z OK 2022-11-23T03:04:48.4288859Z 2022-11-23T03:04:48.4289039Z Generating XML reports... 2022-11-23T03:04:48.4289610Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024722.xml 2022-11-23T03:04:48.4290239Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4290925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4291384Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4291987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4292460Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4292683Z 2022-11-23T03:04:48.4292786Z Running tests... 2022-11-23T03:04:48.4293214Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4293722Z test_tensor_dtype_complex (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80109 2022-11-23T03:04:48.4294225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80110 2022-11-23T03:04:48.4294863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4295320Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4295922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4296399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4296842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4297335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4297987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4298430Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4299031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4299507Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4299956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4300444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4301121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4301830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4302395Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4302985Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4303322Z ok (5.104s) 2022-11-23T03:04:48.4303465Z 2022-11-23T03:04:48.4303746Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4304075Z Ran 1 test in 5.104s 2022-11-23T03:04:48.4304235Z 2022-11-23T03:04:48.4304321Z OK 2022-11-23T03:04:48.4304448Z 2022-11-23T03:04:48.4304567Z Generating XML reports... 2022-11-23T03:04:48.4305112Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024731.xml 2022-11-23T03:04:48.4305733Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4306375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4306835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4307444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4307987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4308214Z 2022-11-23T03:04:48.4308319Z Running tests... 2022-11-23T03:04:48.4308731Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4309237Z test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80321 2022-11-23T03:04:48.4309761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80322 2022-11-23T03:04:48.4310399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4310855Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4311455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4311936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4312363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4312856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4313509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4313961Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4314561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4315040Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4315487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4315961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4316636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4317345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4318410Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:04:48.4319034Z warnings.warn( 2022-11-23T03:04:48.4319947Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:04:48.4320641Z warnings.warn( 2022-11-23T03:04:48.4321567Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:04:48.4322186Z warnings.warn( 2022-11-23T03:04:48.4323086Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:04:48.4323705Z warnings.warn( 2022-11-23T03:04:48.4323933Z ok (5.002s) 2022-11-23T03:04:48.4324079Z 2022-11-23T03:04:48.4324364Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4324700Z Ran 1 test in 5.003s 2022-11-23T03:04:48.4324911Z 2022-11-23T03:04:48.4325003Z OK 2022-11-23T03:04:48.4325135Z 2022-11-23T03:04:48.4325256Z Generating XML reports... 2022-11-23T03:04:48.4325821Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123024740.xml 2022-11-23T03:04:48.4326422Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4327070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4327534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4328139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4328621Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4328851Z 2022-11-23T03:04:48.4328960Z Running tests... 2022-11-23T03:04:48.4329390Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4329902Z test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80530 2022-11-23T03:04:48.4330439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80531 2022-11-23T03:04:48.4331201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4331665Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4332269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4332744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4333190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4333837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4334279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4334883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4335356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4335804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4336300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4336801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4337479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4338168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4338859Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4339394Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4340298Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4341048Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4341945Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4342692Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4343090Z ok (5.265s) 2022-11-23T03:04:48.4343239Z 2022-11-23T03:04:48.4343528Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4343839Z Ran 1 test in 5.266s 2022-11-23T03:04:48.4343994Z 2022-11-23T03:04:48.4344080Z OK 2022-11-23T03:04:48.4344208Z 2022-11-23T03:04:48.4344327Z Generating XML reports... 2022-11-23T03:04:48.4344903Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024749.xml 2022-11-23T03:04:48.4345533Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4346174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4346628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4347212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4347700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4347923Z 2022-11-23T03:04:48.4348025Z Running tests... 2022-11-23T03:04:48.4348452Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4348967Z test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80744 2022-11-23T03:04:48.4349499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80745 2022-11-23T03:04:48.4350136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4350571Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4351173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4351646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4352102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4352748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4353201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4353802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4354275Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4354700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4355189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4355686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4356422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4357139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4357726Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4358240Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4359116Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4359859Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4360756Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4361553Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4361885Z ok (5.213s) 2022-11-23T03:04:48.4362029Z 2022-11-23T03:04:48.4362306Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4362636Z Ran 1 test in 5.213s 2022-11-23T03:04:48.4362792Z 2022-11-23T03:04:48.4362876Z OK 2022-11-23T03:04:48.4363003Z 2022-11-23T03:04:48.4363103Z Generating XML reports... 2022-11-23T03:04:48.4363675Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024758.xml 2022-11-23T03:04:48.4364303Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4364944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4365405Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4366004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4366477Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4366701Z 2022-11-23T03:04:48.4366803Z Running tests... 2022-11-23T03:04:48.4367211Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4367727Z test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80958 2022-11-23T03:04:48.4368253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80959 2022-11-23T03:04:48.4368889Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4369347Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4369954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4370428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4370912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4371567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4372019Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4372621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4373157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4373683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4374356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4374937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4375763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4376620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4377318Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4377942Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4379016Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4379994Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4381065Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4381957Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4382330Z ok (5.117s) 2022-11-23T03:04:48.4382505Z 2022-11-23T03:04:48.4382835Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4383232Z Ran 1 test in 5.118s 2022-11-23T03:04:48.4383417Z 2022-11-23T03:04:48.4383524Z OK 2022-11-23T03:04:48.4383674Z 2022-11-23T03:04:48.4383814Z Generating XML reports... 2022-11-23T03:04:48.4384501Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024808.xml 2022-11-23T03:04:48.4385271Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4386021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4386572Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4387295Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4387867Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4388139Z 2022-11-23T03:04:48.4388262Z Running tests... 2022-11-23T03:04:48.4388771Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4389402Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81172 2022-11-23T03:04:48.4390028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81173 2022-11-23T03:04:48.4390800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4391344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4392063Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4392625Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4393073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4393713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4394167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4394748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4395286Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4395731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4396218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4396713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4397393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4398099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4398664Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4399180Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4400137Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4400876Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4401765Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4402497Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4403382Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4404126Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4405013Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4405741Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4406631Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4407345Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4408236Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4408969Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4409857Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4410638Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4411547Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4412280Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4412685Z ok (5.215s) 2022-11-23T03:04:48.4412831Z 2022-11-23T03:04:48.4413113Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4413443Z Ran 1 test in 5.215s 2022-11-23T03:04:48.4413581Z 2022-11-23T03:04:48.4413668Z OK 2022-11-23T03:04:48.4413794Z 2022-11-23T03:04:48.4413912Z Generating XML reports... 2022-11-23T03:04:48.4414487Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024817.xml 2022-11-23T03:04:48.4415115Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4415755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4416208Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4416806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4417330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4417556Z 2022-11-23T03:04:48.4417656Z Running tests... 2022-11-23T03:04:48.4418085Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4418609Z test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81386 2022-11-23T03:04:48.4419144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81387 2022-11-23T03:04:48.4419782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4420237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4420816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4421297Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4421745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4422395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4422845Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4423447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4423920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4424347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4424842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4425330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4426014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4426717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4427300Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4427825Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4428709Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4429447Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4430371Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4431124Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4432005Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4432736Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4433617Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4434360Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4434750Z ok (5.033s) 2022-11-23T03:04:48.4434893Z 2022-11-23T03:04:48.4435174Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4435504Z Ran 1 test in 5.033s 2022-11-23T03:04:48.4435664Z 2022-11-23T03:04:48.4435732Z OK 2022-11-23T03:04:48.4435863Z 2022-11-23T03:04:48.4435982Z Generating XML reports... 2022-11-23T03:04:48.4436560Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024826.xml 2022-11-23T03:04:48.4437186Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4437829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4438285Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4438879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4439343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4439571Z 2022-11-23T03:04:48.4439672Z Running tests... 2022-11-23T03:04:48.4440102Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4440629Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81600 2022-11-23T03:04:48.4441165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81601 2022-11-23T03:04:48.4441800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4442247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4442825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4443306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4443750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4444396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4444847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4445448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4445917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4446361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4446832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4447325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4448062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4448776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4449360Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4449876Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4450887Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4451628Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4452499Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4453349Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4454237Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4454971Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4455853Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4456591Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4456929Z ok (5.221s) 2022-11-23T03:04:48.4457071Z 2022-11-23T03:04:48.4457352Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4457681Z Ran 1 test in 5.221s 2022-11-23T03:04:48.4457837Z 2022-11-23T03:04:48.4457904Z OK 2022-11-23T03:04:48.4458032Z 2022-11-23T03:04:48.4458150Z Generating XML reports... 2022-11-23T03:04:48.4458723Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024835.xml 2022-11-23T03:04:48.4459352Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4459992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4460444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4461048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4461513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4461739Z 2022-11-23T03:04:48.4461841Z Running tests... 2022-11-23T03:04:48.4462264Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4462774Z test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81814 2022-11-23T03:04:48.4463294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81815 2022-11-23T03:04:48.4463933Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4464385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4464986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4465513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4465966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4466620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4467074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4467674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4468144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4468593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4469066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4469560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4470318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4471026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4471611Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4472132Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4473021Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4473754Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4474621Z /opt/conda/lib/python3.8/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target 2022-11-23T03:04:48.4475357Z warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does ' 2022-11-23T03:04:48.4475690Z ok (5.616s) 2022-11-23T03:04:48.4475831Z 2022-11-23T03:04:48.4476106Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4476431Z Ran 1 test in 5.616s 2022-11-23T03:04:48.4476585Z 2022-11-23T03:04:48.4476669Z OK 2022-11-23T03:04:48.4476799Z 2022-11-23T03:04:48.4476916Z Generating XML reports... 2022-11-23T03:04:48.4477473Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123024844.xml 2022-11-23T03:04:48.4478106Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4478747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4479207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4479810Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4480285Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4480509Z 2022-11-23T03:04:48.4480614Z Running tests... 2022-11-23T03:04:48.4481038Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4481578Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82028 2022-11-23T03:04:48.4482148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82029 2022-11-23T03:04:48.4482783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4483303Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4483914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4484387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4484830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4485453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4485913Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4486511Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4486982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4487422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4487983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zjlsm81 2022-11-23T03:04:48.4488518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zjlsm81/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4489051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpys0w9gh0 2022-11-23T03:04:48.4489565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpys0w9gh0/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4490145Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4490749Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4491212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4491697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4492054Z ok (7.506s) 2022-11-23T03:04:48.4492200Z 2022-11-23T03:04:48.4492463Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4492797Z Ran 1 test in 7.506s 2022-11-23T03:04:48.4492953Z 2022-11-23T03:04:48.4493039Z OK 2022-11-23T03:04:48.4493168Z 2022-11-23T03:04:48.4493288Z Generating XML reports... 2022-11-23T03:04:48.4493929Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024853.xml 2022-11-23T03:04:48.4494600Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4495243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4495676Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4496275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4496757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4496985Z 2022-11-23T03:04:48.4497087Z Running tests... 2022-11-23T03:04:48.4497516Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4498094Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82250 2022-11-23T03:04:48.4498683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82251 2022-11-23T03:04:48.4499317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4499751Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4500354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4500829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4501404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4502191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4502393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4502869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4503095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4503368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4503665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bffe5wb 2022-11-23T03:04:48.4503977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bffe5wb/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4504392Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4504668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgktjawmh 2022-11-23T03:04:48.4504981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgktjawmh/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4505330Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4505603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4505871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4505983Z ok (7.324s) 2022-11-23T03:04:48.4505990Z 2022-11-23T03:04:48.4506330Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4506463Z Ran 1 test in 7.325s 2022-11-23T03:04:48.4506470Z 2022-11-23T03:04:48.4506570Z OK 2022-11-23T03:04:48.4506577Z 2022-11-23T03:04:48.4506718Z Generating XML reports... 2022-11-23T03:04:48.4507289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024905.xml 2022-11-23T03:04:48.4507679Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4508143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4508345Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4508823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4509048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4509054Z 2022-11-23T03:04:48.4509174Z Running tests... 2022-11-23T03:04:48.4509510Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4509917Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82472 2022-11-23T03:04:48.4510178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82473 2022-11-23T03:04:48.4510641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4510831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4511314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4511536Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4511803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4512266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4512471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4513020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4513232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4513455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4513700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m2ayf_v 2022-11-23T03:04:48.4513960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m2ayf_v/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4514203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkm6nv5l 2022-11-23T03:04:48.4514458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkm6nv5l/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4514749Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4515085Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4515319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4515550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4515644Z ok (7.427s) 2022-11-23T03:04:48.4515651Z 2022-11-23T03:04:48.4515929Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4516035Z Ran 1 test in 7.427s 2022-11-23T03:04:48.4516041Z 2022-11-23T03:04:48.4516125Z OK 2022-11-23T03:04:48.4516131Z 2022-11-23T03:04:48.4516231Z Generating XML reports... 2022-11-23T03:04:48.4516693Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024916.xml 2022-11-23T03:04:48.4517016Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4517408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4517583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4517977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4518163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4518169Z 2022-11-23T03:04:48.4518271Z Running tests... 2022-11-23T03:04:48.4518547Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4518894Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82694 2022-11-23T03:04:48.4519110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82695 2022-11-23T03:04:48.4519497Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4519673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4520066Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4520251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4520475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4520860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4521031Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4521427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4521614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4521896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4522147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbha9r2td 2022-11-23T03:04:48.4522387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbha9r2td/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4522677Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4522927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4rhsexss 2022-11-23T03:04:48.4523185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4rhsexss/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4523469Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4523695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4523922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4524078Z ok (7.119s) 2022-11-23T03:04:48.4524088Z 2022-11-23T03:04:48.4524371Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4524478Z Ran 1 test in 7.119s 2022-11-23T03:04:48.4524483Z 2022-11-23T03:04:48.4524571Z OK 2022-11-23T03:04:48.4524577Z 2022-11-23T03:04:48.4524697Z Generating XML reports... 2022-11-23T03:04:48.4525156Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024927.xml 2022-11-23T03:04:48.4525480Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4525867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4526043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4526440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4526636Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4526642Z 2022-11-23T03:04:48.4526743Z Running tests... 2022-11-23T03:04:48.4527022Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4527357Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82916 2022-11-23T03:04:48.4527553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82917 2022-11-23T03:04:48.4527938Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4528110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4528509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4528693Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4528923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4529307Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4529479Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4529875Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4530061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4530288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4531039Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4531594Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4531846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsh6asy7y 2022-11-23T03:04:48.4532104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsh6asy7y/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4532346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwkj0v9e 2022-11-23T03:04:48.4532607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwkj0v9e/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4532962Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4533248Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4533344Z ok (5.816s) 2022-11-23T03:04:48.4533350Z 2022-11-23T03:04:48.4533627Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4533732Z Ran 1 test in 5.816s 2022-11-23T03:04:48.4533738Z 2022-11-23T03:04:48.4533826Z OK 2022-11-23T03:04:48.4533831Z 2022-11-23T03:04:48.4533949Z Generating XML reports... 2022-11-23T03:04:48.4534417Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024938.xml 2022-11-23T03:04:48.4534745Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4535111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4535287Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4535690Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4535880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4535885Z 2022-11-23T03:04:48.4535990Z Running tests... 2022-11-23T03:04:48.4536266Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4536595Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83136 2022-11-23T03:04:48.4536814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83137 2022-11-23T03:04:48.4537202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4537376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4537778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4537967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4538192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4538730Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4539117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4539288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4539737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4539933Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4540158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4540696Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4540943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1sumqyh 2022-11-23T03:04:48.4541201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1sumqyh/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4541446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdum9vvpo 2022-11-23T03:04:48.4541753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdum9vvpo/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4542050Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4542336Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4542431Z ok (5.111s) 2022-11-23T03:04:48.4542437Z 2022-11-23T03:04:48.4542692Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4542802Z Ran 1 test in 5.112s 2022-11-23T03:04:48.4542808Z 2022-11-23T03:04:48.4542893Z OK 2022-11-23T03:04:48.4542898Z 2022-11-23T03:04:48.4543017Z Generating XML reports... 2022-11-23T03:04:48.4543480Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024948.xml 2022-11-23T03:04:48.4543809Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4544204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4544376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4544773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4544961Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4544967Z 2022-11-23T03:04:48.4545073Z Running tests... 2022-11-23T03:04:48.4545352Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4545682Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83356 2022-11-23T03:04:48.4545901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83357 2022-11-23T03:04:48.4546290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4546470Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4546867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4547051Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4547279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4547665Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4547834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4548231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4548398Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4548688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4548941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph331je7k 2022-11-23T03:04:48.4549197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph331je7k/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4549440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5rutfxm8 2022-11-23T03:04:48.4549698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5rutfxm8/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4549991Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4550279Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4550378Z ok (5.214s) 2022-11-23T03:04:48.4550384Z 2022-11-23T03:04:48.4550659Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4550765Z Ran 1 test in 5.215s 2022-11-23T03:04:48.4550817Z 2022-11-23T03:04:48.4550906Z OK 2022-11-23T03:04:48.4550915Z 2022-11-23T03:04:48.4551036Z Generating XML reports... 2022-11-23T03:04:48.4551504Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123024957.xml 2022-11-23T03:04:48.4551829Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4552213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4552386Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4552784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4552969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4552975Z 2022-11-23T03:04:48.4553078Z Running tests... 2022-11-23T03:04:48.4553334Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4553687Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83576 2022-11-23T03:04:48.4553904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83577 2022-11-23T03:04:48.4554294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4554466Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4554866Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4555053Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4555282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4555671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4555846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4556242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4556426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4556648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4556897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphpg8uizz 2022-11-23T03:04:48.4557158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphpg8uizz/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4557400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5_579x3 2022-11-23T03:04:48.4557652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5_579x3/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4558017Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4558311Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4558406Z ok (5.842s) 2022-11-23T03:04:48.4558413Z 2022-11-23T03:04:48.4558686Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4558792Z Ran 1 test in 5.842s 2022-11-23T03:04:48.4558798Z 2022-11-23T03:04:48.4558866Z OK 2022-11-23T03:04:48.4558892Z 2022-11-23T03:04:48.4558992Z Generating XML reports... 2022-11-23T03:04:48.4559460Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025006.xml 2022-11-23T03:04:48.4559783Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4560167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4560394Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4560797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4560986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4560992Z 2022-11-23T03:04:48.4561096Z Running tests... 2022-11-23T03:04:48.4561371Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4561692Z test_channels_last_contig (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83796 2022-11-23T03:04:48.4561908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83797 2022-11-23T03:04:48.4562298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4562471Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4562871Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4563057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4563284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4563671Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4563844Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4564229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4590645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4590917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4591277Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4591583Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4591692Z ok (5.194s) 2022-11-23T03:04:48.4591698Z 2022-11-23T03:04:48.4591988Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4592099Z Ran 1 test in 5.195s 2022-11-23T03:04:48.4592105Z 2022-11-23T03:04:48.4592201Z OK 2022-11-23T03:04:48.4592207Z 2022-11-23T03:04:48.4592327Z Generating XML reports... 2022-11-23T03:04:48.4592824Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025016.xml 2022-11-23T03:04:48.4593155Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4593550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4593723Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4594258Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4594455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4594462Z 2022-11-23T03:04:48.4594565Z Running tests... 2022-11-23T03:04:48.4594854Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4595053Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4595549Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84008 2022-11-23T03:04:48.4595768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84009 2022-11-23T03:04:48.4596159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4596400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4596803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4596991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4597216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4597606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4597778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4598173Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4598360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4598584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4598840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97nc3oiv 2022-11-23T03:04:48.4599098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97nc3oiv/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4599343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrw6vlp6 2022-11-23T03:04:48.4599614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrw6vlp6/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4599913Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4600202Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4600301Z ok (7.326s) 2022-11-23T03:04:48.4600307Z 2022-11-23T03:04:48.4600583Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4600671Z Ran 1 test in 7.326s 2022-11-23T03:04:48.4600697Z 2022-11-23T03:04:48.4600769Z OK 2022-11-23T03:04:48.4600774Z 2022-11-23T03:04:48.4600897Z Generating XML reports... 2022-11-23T03:04:48.4601362Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025025.xml 2022-11-23T03:04:48.4601686Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4602072Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4602244Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4602641Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4602830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4602836Z 2022-11-23T03:04:48.4602939Z Running tests... 2022-11-23T03:04:48.4603216Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4603505Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4603832Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84230 2022-11-23T03:04:48.4604048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84231 2022-11-23T03:04:48.4604441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4604614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4605014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4605199Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4605426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4605865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4606039Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4606436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4606603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4606827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4607074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cwheqlx 2022-11-23T03:04:48.4607335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cwheqlx/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4607580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfptrvmxw 2022-11-23T03:04:48.4607846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfptrvmxw/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4608139Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4608423Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4608519Z ok (7.215s) 2022-11-23T03:04:48.4608525Z 2022-11-23T03:04:48.4608799Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4608910Z Ran 1 test in 7.216s 2022-11-23T03:04:48.4608916Z 2022-11-23T03:04:48.4609003Z OK 2022-11-23T03:04:48.4609009Z 2022-11-23T03:04:48.4609127Z Generating XML reports... 2022-11-23T03:04:48.4609596Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025037.xml 2022-11-23T03:04:48.4609920Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4610310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4610487Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4610956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4611144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4611150Z 2022-11-23T03:04:48.4611253Z Running tests... 2022-11-23T03:04:48.4611535Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4611746Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4612057Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84452 2022-11-23T03:04:48.4612274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84453 2022-11-23T03:04:48.4612727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4612907Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4613311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4613497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4613722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4614109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4614281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4614677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4614863Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4615149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4615401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbuc6wqgw 2022-11-23T03:04:48.4615660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbuc6wqgw/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4615902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpao20553h 2022-11-23T03:04:48.4616156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpao20553h/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4616451Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4616735Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4616964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4617194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4617411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4617642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4618611Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:04:48.4618715Z warnings.warn( 2022-11-23T03:04:48.4618944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4619909Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:04:48.4620018Z warnings.warn( 2022-11-23T03:04:48.4620245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4620473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4620701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4620927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4621153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4621378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4621664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4621763Z ok (7.519s) 2022-11-23T03:04:48.4621769Z 2022-11-23T03:04:48.4622054Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4622161Z Ran 1 test in 7.519s 2022-11-23T03:04:48.4622167Z 2022-11-23T03:04:48.4622255Z OK 2022-11-23T03:04:48.4622261Z 2022-11-23T03:04:48.4622381Z Generating XML reports... 2022-11-23T03:04:48.4622846Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025048.xml 2022-11-23T03:04:48.4623169Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4623557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4623732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4624134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4624375Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4624381Z 2022-11-23T03:04:48.4624466Z Running tests... 2022-11-23T03:04:48.4624746Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4624977Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4625286Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84674 2022-11-23T03:04:48.4625502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84675 2022-11-23T03:04:48.4625893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4626065Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4626468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4626655Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4626927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4627385Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4627588Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4628056Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4628278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4628547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4628842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpig6uho8s 2022-11-23T03:04:48.4629158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpig6uho8s/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4629505Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4629800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptniejvgo 2022-11-23T03:04:48.4630112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptniejvgo/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4630458Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4630711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4630983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4631258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4631615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4632784Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:04:48.4632909Z warnings.warn( 2022-11-23T03:04:48.4633184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4634341Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:04:48.4634531Z warnings.warn( 2022-11-23T03:04:48.4634807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4635080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4635354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4635629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4635900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4636168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4636444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4636566Z ok (7.316s) 2022-11-23T03:04:48.4636573Z 2022-11-23T03:04:48.4636917Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4637044Z Ran 1 test in 7.316s 2022-11-23T03:04:48.4637050Z 2022-11-23T03:04:48.4637158Z OK 2022-11-23T03:04:48.4637165Z 2022-11-23T03:04:48.4637304Z Generating XML reports... 2022-11-23T03:04:48.4637868Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025059.xml 2022-11-23T03:04:48.4638261Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4638732Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4638940Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4639424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4639628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4639663Z 2022-11-23T03:04:48.4639774Z Running tests... 2022-11-23T03:04:48.4640107Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4640412Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4640973Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84896 2022-11-23T03:04:48.4641238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84897 2022-11-23T03:04:48.4641703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4641912Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4642393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4642619Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4642949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4643412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4643586Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4643983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4644168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4644392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4644640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjyfy5e7b 2022-11-23T03:04:48.4644899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjyfy5e7b/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4645243Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4645491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdr330msv 2022-11-23T03:04:48.4645751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdr330msv/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4646018Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4646249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4646476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4646709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4646937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4647034Z ok (7.310s) 2022-11-23T03:04:48.4647040Z 2022-11-23T03:04:48.4647330Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4647440Z Ran 1 test in 7.310s 2022-11-23T03:04:48.4647446Z 2022-11-23T03:04:48.4647532Z OK 2022-11-23T03:04:48.4647537Z 2022-11-23T03:04:48.4647656Z Generating XML reports... 2022-11-23T03:04:48.4648117Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025111.xml 2022-11-23T03:04:48.4648441Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4648831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4649003Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4649401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4649588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4649598Z 2022-11-23T03:04:48.4649704Z Running tests... 2022-11-23T03:04:48.4649985Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4650231Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4650858Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85118 2022-11-23T03:04:48.4651090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85119 2022-11-23T03:04:48.4651486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4651656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4652053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4652312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4652548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4652920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4653095Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4653489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4653678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4653903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4654148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6px791k2 2022-11-23T03:04:48.4654404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6px791k2/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4654705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpctrjn67_ 2022-11-23T03:04:48.4654964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpctrjn67_/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4655260Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4655546Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4655775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4656001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4656230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4656456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4656555Z ok (7.312s) 2022-11-23T03:04:48.4656564Z 2022-11-23T03:04:48.4656844Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4656950Z Ran 1 test in 7.313s 2022-11-23T03:04:48.4656955Z 2022-11-23T03:04:48.4657041Z OK 2022-11-23T03:04:48.4657047Z 2022-11-23T03:04:48.4657169Z Generating XML reports... 2022-11-23T03:04:48.4657629Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025122.xml 2022-11-23T03:04:48.4657935Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4658321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4658495Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4658892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4659079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4659089Z 2022-11-23T03:04:48.4659194Z Running tests... 2022-11-23T03:04:48.4659469Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4659702Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4660188Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85340 2022-11-23T03:04:48.4660404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85341 2022-11-23T03:04:48.4660788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4660960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4661356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4661598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4661828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4662216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4662388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4662782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4662967Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4663196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4663442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmxyc0g0 2022-11-23T03:04:48.4663700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmxyc0g0/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4663973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4vtkc18 2022-11-23T03:04:48.4664234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4vtkc18/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4664528Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4664813Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4665041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4665267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4666048Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:04:48.4666827Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:04:48.4667081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4667290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4667394Z ok (7.447s) 2022-11-23T03:04:48.4667400Z 2022-11-23T03:04:48.4667679Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4667787Z Ran 1 test in 7.447s 2022-11-23T03:04:48.4667793Z 2022-11-23T03:04:48.4667881Z OK 2022-11-23T03:04:48.4667886Z 2022-11-23T03:04:48.4668005Z Generating XML reports... 2022-11-23T03:04:48.4668469Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025133.xml 2022-11-23T03:04:48.4668792Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4669179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4669352Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4669746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4669991Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4669998Z 2022-11-23T03:04:48.4670104Z Running tests... 2022-11-23T03:04:48.4670384Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4670617Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4671104Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85562 2022-11-23T03:04:48.4671323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85563 2022-11-23T03:04:48.4671713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4671884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4672282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4672515Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4672720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4673110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4673282Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4673679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4673865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4674089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4674340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8luznp9 2022-11-23T03:04:48.4674605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8luznp9/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4674895Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4675142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1vu90id 2022-11-23T03:04:48.4675400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1vu90id/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4675688Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4675785Z ok (7.128s) 2022-11-23T03:04:48.4675791Z 2022-11-23T03:04:48.4676067Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4676173Z Ran 1 test in 7.128s 2022-11-23T03:04:48.4676179Z 2022-11-23T03:04:48.4676267Z OK 2022-11-23T03:04:48.4676273Z 2022-11-23T03:04:48.4676395Z Generating XML reports... 2022-11-23T03:04:48.4676860Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025145.xml 2022-11-23T03:04:48.4677187Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4677572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4677744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4678119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4678308Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4678313Z 2022-11-23T03:04:48.4678420Z Running tests... 2022-11-23T03:04:48.4678694Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4678919Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4679303Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85784 2022-11-23T03:04:48.4679526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85785 2022-11-23T03:04:48.4679917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4680089Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4680483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4680670Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4680897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4681271Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4681501Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4681898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4682085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4682311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4682560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmchiojmn 2022-11-23T03:04:48.4682820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmchiojmn/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4683064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2sz16bc 2022-11-23T03:04:48.4683321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2sz16bc/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4683613Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4683885Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4684116Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4684343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4684571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4684799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4684896Z ok (7.417s) 2022-11-23T03:04:48.4684902Z 2022-11-23T03:04:48.4685178Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4685281Z Ran 1 test in 7.417s 2022-11-23T03:04:48.4685287Z 2022-11-23T03:04:48.4685362Z OK 2022-11-23T03:04:48.4685367Z 2022-11-23T03:04:48.4685475Z Generating XML reports... 2022-11-23T03:04:48.4685924Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025156.xml 2022-11-23T03:04:48.4686241Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4686613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4686773Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4687159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4687343Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4687349Z 2022-11-23T03:04:48.4687446Z Running tests... 2022-11-23T03:04:48.4687706Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4687944Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4688353Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86006 2022-11-23T03:04:48.4688564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86007 2022-11-23T03:04:48.4688937Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4689109Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4689507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4689695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4689917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4690297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4690590Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4690981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4691168Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4691394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4691640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1av1l3e 2022-11-23T03:04:48.4691899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1av1l3e/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4692142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoeq1gvc6 2022-11-23T03:04:48.4692393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoeq1gvc6/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4692688Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4692972Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4693752Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:04:48.4694530Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:04:48.4695489Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:04:48.4695594Z warnings.warn( 2022-11-23T03:04:48.4695822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4696837Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:04:48.4696949Z warnings.warn( 2022-11-23T03:04:48.4697178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4697404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4697626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4697716Z ok (7.510s) 2022-11-23T03:04:48.4697722Z 2022-11-23T03:04:48.4697991Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4698093Z Ran 1 test in 7.511s 2022-11-23T03:04:48.4698099Z 2022-11-23T03:04:48.4698186Z OK 2022-11-23T03:04:48.4698191Z 2022-11-23T03:04:48.4698309Z Generating XML reports... 2022-11-23T03:04:48.4698841Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025207.xml 2022-11-23T03:04:48.4699167Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4699553Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4699706Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4700103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4700292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4700297Z 2022-11-23T03:04:48.4700401Z Running tests... 2022-11-23T03:04:48.4700681Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4700929Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4701261Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86228 2022-11-23T03:04:48.4701478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86229 2022-11-23T03:04:48.4701860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4702025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4702416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4702598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4702822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4703199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4703378Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4703773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4703958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4704171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4704405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9uzmjc98 2022-11-23T03:04:48.4704651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9uzmjc98/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4704883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmd3tk6sy 2022-11-23T03:04:48.4705127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmd3tk6sy/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4705518Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4705795Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4706735Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:04:48.4706838Z warnings.warn( 2022-11-23T03:04:48.4707054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4708000Z /opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-11-23T03:04:48.4708143Z warnings.warn( 2022-11-23T03:04:48.4708361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4708575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4708791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4708885Z ok (7.220s) 2022-11-23T03:04:48.4708891Z 2022-11-23T03:04:48.4709169Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4709266Z Ran 1 test in 7.220s 2022-11-23T03:04:48.4709272Z 2022-11-23T03:04:48.4709355Z OK 2022-11-23T03:04:48.4709361Z 2022-11-23T03:04:48.4709484Z Generating XML reports... 2022-11-23T03:04:48.4709950Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025218.xml 2022-11-23T03:04:48.4710282Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4710672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4710850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4711251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4711439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4711445Z 2022-11-23T03:04:48.4711552Z Running tests... 2022-11-23T03:04:48.4711821Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4712068Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4712364Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86450 2022-11-23T03:04:48.4712586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86451 2022-11-23T03:04:48.4712953Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4713132Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4713533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4713719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4713947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4714334Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4714507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4714960Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4715155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4715386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4715636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tl1u425 2022-11-23T03:04:48.4715885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu56xbi35 2022-11-23T03:04:48.4716143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tl1u425/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4716401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu56xbi35/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4716704Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4717045Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4717273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4717509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4717739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4717968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4718045Z ok (7.415s) 2022-11-23T03:04:48.4718071Z 2022-11-23T03:04:48.4718328Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4718440Z Ran 1 test in 7.416s 2022-11-23T03:04:48.4718446Z 2022-11-23T03:04:48.4718536Z OK 2022-11-23T03:04:48.4718542Z 2022-11-23T03:04:48.4718660Z Generating XML reports... 2022-11-23T03:04:48.4719121Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025230.xml 2022-11-23T03:04:48.4719450Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4719840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4720016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4720417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4720608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4720614Z 2022-11-23T03:04:48.4720719Z Running tests... 2022-11-23T03:04:48.4720994Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4721246Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4721546Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86672 2022-11-23T03:04:48.4721768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86673 2022-11-23T03:04:48.4722156Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4722324Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4722718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4722900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4723120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4723507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4723659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4724118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4724315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4724542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4724784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptc16yywf 2022-11-23T03:04:48.4725044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptc16yywf/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4725287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxmzve04 2022-11-23T03:04:48.4725545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxmzve04/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4725840Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4726168Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4726394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4726628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4726856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4727082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4727310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4727535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4727764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4727991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4728090Z ok (7.412s) 2022-11-23T03:04:48.4728099Z 2022-11-23T03:04:48.4728376Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4728464Z Ran 1 test in 7.412s 2022-11-23T03:04:48.4728494Z 2022-11-23T03:04:48.4728561Z OK 2022-11-23T03:04:48.4728567Z 2022-11-23T03:04:48.4728690Z Generating XML reports... 2022-11-23T03:04:48.4729150Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025241.xml 2022-11-23T03:04:48.4729474Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4729862Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4730040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4730427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4730769Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4730775Z 2022-11-23T03:04:48.4730883Z Running tests... 2022-11-23T03:04:48.4731168Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4731507Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86894 2022-11-23T03:04:48.4731723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86895 2022-11-23T03:04:48.4732105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4732276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4732670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4732854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4733144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4733538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4733714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4734116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4734305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4734509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4734758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8585mzd 2022-11-23T03:04:48.4735003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8585mzd/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4735313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0crtvqe8 2022-11-23T03:04:48.4735573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0crtvqe8/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4735863Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4736136Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4736226Z ok (5.206s) 2022-11-23T03:04:48.4736233Z 2022-11-23T03:04:48.4736494Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4736594Z Ran 1 test in 5.206s 2022-11-23T03:04:48.4736600Z 2022-11-23T03:04:48.4736691Z OK 2022-11-23T03:04:48.4736697Z 2022-11-23T03:04:48.4736817Z Generating XML reports... 2022-11-23T03:04:48.4737286Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025253.xml 2022-11-23T03:04:48.4737616Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4738006Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4738176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4738570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4738754Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4738760Z 2022-11-23T03:04:48.4738862Z Running tests... 2022-11-23T03:04:48.4739138Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4739489Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87114 2022-11-23T03:04:48.4739712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87115 2022-11-23T03:04:48.4740085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4740255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4740645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4740830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4741054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4741440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4741606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4741995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4742245Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4742478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4742727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwx1vtb8 2022-11-23T03:04:48.4742987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwx1vtb8/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4743232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgg8cyvj 2022-11-23T03:04:48.4743497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgg8cyvj/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4743791Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4744078Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4744170Z ok (5.208s) 2022-11-23T03:04:48.4744176Z 2022-11-23T03:04:48.4744501Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4744611Z Ran 1 test in 5.208s 2022-11-23T03:04:48.4744617Z 2022-11-23T03:04:48.4744708Z OK 2022-11-23T03:04:48.4744714Z 2022-11-23T03:04:48.4744813Z Generating XML reports... 2022-11-23T03:04:48.4745273Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025302.xml 2022-11-23T03:04:48.4745604Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4745992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4746167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4746558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4746741Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4746750Z 2022-11-23T03:04:48.4746858Z Running tests... 2022-11-23T03:04:48.4747136Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4747498Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87334 2022-11-23T03:04:48.4747719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87335 2022-11-23T03:04:48.4748106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4748284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4748686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4748869Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4749103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4749488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4749666Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4750050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4750232Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4750463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4750709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeq_c9txx 2022-11-23T03:04:48.4750947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeq_c9txx/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4751190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14h6cjuz 2022-11-23T03:04:48.4751504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14h6cjuz/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4751803Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4752090Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4752186Z ok (5.125s) 2022-11-23T03:04:48.4752192Z 2022-11-23T03:04:48.4752471Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4752572Z Ran 1 test in 5.125s 2022-11-23T03:04:48.4752578Z 2022-11-23T03:04:48.4752668Z OK 2022-11-23T03:04:48.4752674Z 2022-11-23T03:04:48.4752797Z Generating XML reports... 2022-11-23T03:04:48.4753268Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025311.xml 2022-11-23T03:04:48.4753627Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4754159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4754368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4754851Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4755076Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4755083Z 2022-11-23T03:04:48.4755208Z Running tests... 2022-11-23T03:04:48.4755546Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4755816Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4756229Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87554 2022-11-23T03:04:48.4756496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87555 2022-11-23T03:04:48.4756964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4757145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4757621Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4757842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4758113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4758576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4758785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4759262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4759476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4759748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4760045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1a2ii2a 2022-11-23T03:04:48.4760357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1a2ii2a/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4760657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9n_e_ige 2022-11-23T03:04:48.4760960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9n_e_ige/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4761305Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4761643Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4761762Z ok (5.305s) 2022-11-23T03:04:48.4761774Z 2022-11-23T03:04:48.4762152Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4762280Z Ran 1 test in 5.306s 2022-11-23T03:04:48.4762287Z 2022-11-23T03:04:48.4762396Z OK 2022-11-23T03:04:48.4762403Z 2022-11-23T03:04:48.4762545Z Generating XML reports... 2022-11-23T03:04:48.4763085Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025320.xml 2022-11-23T03:04:48.4763484Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4763897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4764072Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4764470Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4764662Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4764724Z 2022-11-23T03:04:48.4764828Z Running tests... 2022-11-23T03:04:48.4765110Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4765337Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4765679Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87774 2022-11-23T03:04:48.4765895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87775 2022-11-23T03:04:48.4766282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4766454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4766847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4767043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4767270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4767656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4767830Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4768214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4768403Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4768631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4768878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbciyxx1 2022-11-23T03:04:48.4769143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbciyxx1/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4769374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0eih8vri 2022-11-23T03:04:48.4769630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0eih8vri/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4769919Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4770210Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4770307Z ok (5.208s) 2022-11-23T03:04:48.4770313Z 2022-11-23T03:04:48.4770647Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4770758Z Ran 1 test in 5.209s 2022-11-23T03:04:48.4770764Z 2022-11-23T03:04:48.4770853Z OK 2022-11-23T03:04:48.4770859Z 2022-11-23T03:04:48.4770980Z Generating XML reports... 2022-11-23T03:04:48.4771448Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025329.xml 2022-11-23T03:04:48.4771862Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4772260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4772431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4772830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4773026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4773032Z 2022-11-23T03:04:48.4773140Z Running tests... 2022-11-23T03:04:48.4773422Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4773759Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87994 2022-11-23T03:04:48.4774045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87995 2022-11-23T03:04:48.4774437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4774598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4774974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4775164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4775393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4775777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4775948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4776343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4776530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4776757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4776906Z skip: Need at least 4 CUDA devices (4.710s) 2022-11-23T03:04:48.4776913Z 2022-11-23T03:04:48.4777188Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4777298Z Ran 1 test in 4.710s 2022-11-23T03:04:48.4777303Z 2022-11-23T03:04:48.4777414Z OK (skipped=1) 2022-11-23T03:04:48.4777419Z 2022-11-23T03:04:48.4777539Z Generating XML reports... 2022-11-23T03:04:48.4778007Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025339.xml 2022-11-23T03:04:48.4778331Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4778707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4778884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4779283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4779474Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4779480Z 2022-11-23T03:04:48.4779586Z Running tests... 2022-11-23T03:04:48.4779862Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4780039Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.4780352Z Tests that DDP with ``device_ids`` specified can run a forward and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88195 2022-11-23T03:04:48.4780569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88196 2022-11-23T03:04:48.4781009Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4781186Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4781588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4781773Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4782000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4782237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.4782623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4782786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4783183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4783426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4783651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4783886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.4784301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4784708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.4784954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtvj8zm9 2022-11-23T03:04:48.4785214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtvj8zm9/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4785458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxccm050 2022-11-23T03:04:48.4785724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxccm050/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4786011Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4786300Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4786376Z ok (8.796s) 2022-11-23T03:04:48.4786382Z 2022-11-23T03:04:48.4786660Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4786766Z Ran 1 test in 8.796s 2022-11-23T03:04:48.4786772Z 2022-11-23T03:04:48.4786860Z OK 2022-11-23T03:04:48.4786866Z 2022-11-23T03:04:48.4786988Z Generating XML reports... 2022-11-23T03:04:48.4787456Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025347.xml 2022-11-23T03:04:48.4787789Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4788183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4788361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4788765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4788956Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4788962Z 2022-11-23T03:04:48.4789066Z Running tests... 2022-11-23T03:04:48.4789345Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4789669Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88417 2022-11-23T03:04:48.4789889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88418 2022-11-23T03:04:48.4790333Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4790513Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4790901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4791091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4791322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4791707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4791859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4792255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4792440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4792726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4792976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1xfujtb 2022-11-23T03:04:48.4793239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1xfujtb/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4793481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvou1l2s 2022-11-23T03:04:48.4793736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvou1l2s/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4794033Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4794322Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4794559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4794781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4795012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4795239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4795466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4795693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4795921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4796150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4796250Z ok (5.321s) 2022-11-23T03:04:48.4796255Z 2022-11-23T03:04:48.4796536Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4796624Z Ran 1 test in 5.322s 2022-11-23T03:04:48.4796648Z 2022-11-23T03:04:48.4796719Z OK 2022-11-23T03:04:48.4796725Z 2022-11-23T03:04:48.4796848Z Generating XML reports... 2022-11-23T03:04:48.4797310Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025400.xml 2022-11-23T03:04:48.4797636Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4798027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4798199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4798597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4798778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4798783Z 2022-11-23T03:04:48.4798885Z Running tests... 2022-11-23T03:04:48.4799165Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4799546Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88637 2022-11-23T03:04:48.4799770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88638 2022-11-23T03:04:48.4800158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4800331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4800726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4800909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4801137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4801718Z /opt/conda/lib/python3.8/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-11-23T03:04:48.4802062Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-11-23T03:04:48.4802314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf02qi7ce 2022-11-23T03:04:48.4802576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf02qi7ce/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4802959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4803112Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4803507Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4803696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4803920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4804492Z /opt/conda/lib/python3.8/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-11-23T03:04:48.4804776Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-11-23T03:04:48.4805026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdll_2oh0 2022-11-23T03:04:48.4805282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdll_2oh0/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4805380Z ok (4.611s) 2022-11-23T03:04:48.4805386Z 2022-11-23T03:04:48.4805664Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4805768Z Ran 1 test in 4.612s 2022-11-23T03:04:48.4805775Z 2022-11-23T03:04:48.4805861Z OK 2022-11-23T03:04:48.4805866Z 2022-11-23T03:04:48.4805987Z Generating XML reports... 2022-11-23T03:04:48.4806453Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025409.xml 2022-11-23T03:04:48.4806784Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4807171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4807331Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4807731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4807920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4807926Z 2022-11-23T03:04:48.4808031Z Running tests... 2022-11-23T03:04:48.4808305Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4808685Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88842 2022-11-23T03:04:48.4808915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88843 2022-11-23T03:04:48.4809287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4809461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4809859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4810044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4810274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4810719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4810898Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4811366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4811550Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4811772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4812023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cqsg4k_ 2022-11-23T03:04:48.4812284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cqsg4k_/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4812572Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4812821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptq7iog8d 2022-11-23T03:04:48.4813079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptq7iog8d/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4813369Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4813475Z ok (5.132s) 2022-11-23T03:04:48.4813480Z 2022-11-23T03:04:48.4813758Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4813871Z Ran 1 test in 5.133s 2022-11-23T03:04:48.4813877Z 2022-11-23T03:04:48.4813967Z OK 2022-11-23T03:04:48.4813973Z 2022-11-23T03:04:48.4814090Z Generating XML reports... 2022-11-23T03:04:48.4814533Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025418.xml 2022-11-23T03:04:48.4814859Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4815248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4815421Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4815813Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4816009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4816015Z 2022-11-23T03:04:48.4816118Z Running tests... 2022-11-23T03:04:48.4816392Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4816733Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89062 2022-11-23T03:04:48.4816938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89063 2022-11-23T03:04:48.4817330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4817507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4817906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4818152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4818382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4818773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4818943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4819337Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4819510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4819734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4819980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp623zhtvw 2022-11-23T03:04:48.4820235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp623zhtvw/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4820521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtj86vpq 2022-11-23T03:04:48.4820783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtj86vpq/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4821078Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4821370Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4821468Z ok (5.258s) 2022-11-23T03:04:48.4821473Z 2022-11-23T03:04:48.4821753Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4821862Z Ran 1 test in 5.258s 2022-11-23T03:04:48.4821867Z 2022-11-23T03:04:48.4821965Z OK 2022-11-23T03:04:48.4821970Z 2022-11-23T03:04:48.4822091Z Generating XML reports... 2022-11-23T03:04:48.4822556Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025427.xml 2022-11-23T03:04:48.4822888Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4823273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4823445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4823837Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4824027Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4824033Z 2022-11-23T03:04:48.4824139Z Running tests... 2022-11-23T03:04:48.4824423Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4824737Z test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89282 2022-11-23T03:04:48.4824952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89283 2022-11-23T03:04:48.4825345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4825521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4825895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4826088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4826319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4826705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4826875Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4827269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4827510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4827738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4827986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwsa9dqc 2022-11-23T03:04:48.4828248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwsa9dqc/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4828493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyt9y2f0k 2022-11-23T03:04:48.4828754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyt9y2f0k/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4829049Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4829336Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4829568Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4829850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4830079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4830309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4830405Z ok (7.226s) 2022-11-23T03:04:48.4830411Z 2022-11-23T03:04:48.4830693Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4830781Z Ran 1 test in 7.226s 2022-11-23T03:04:48.4830809Z 2022-11-23T03:04:48.4830876Z OK 2022-11-23T03:04:48.4830882Z 2022-11-23T03:04:48.4831002Z Generating XML reports... 2022-11-23T03:04:48.4831469Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025436.xml 2022-11-23T03:04:48.4831791Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4832182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4832356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4832755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4832948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4832954Z 2022-11-23T03:04:48.4833054Z Running tests... 2022-11-23T03:04:48.4833332Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4834331Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82632 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.638s) 2022-11-23T03:04:48.4834341Z 2022-11-23T03:04:48.4834619Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4834728Z Ran 1 test in 0.638s 2022-11-23T03:04:48.4834733Z 2022-11-23T03:04:48.4834841Z OK (skipped=1) 2022-11-23T03:04:48.4834846Z 2022-11-23T03:04:48.4834968Z Generating XML reports... 2022-11-23T03:04:48.4835433Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025447.xml 2022-11-23T03:04:48.4835762Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4836134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4836313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4836772Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4836968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4836974Z 2022-11-23T03:04:48.4837080Z Running tests... 2022-11-23T03:04:48.4837362Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4838350Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/83301 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.617s) 2022-11-23T03:04:48.4838357Z 2022-11-23T03:04:48.4838629Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4838737Z Ran 1 test in 0.618s 2022-11-23T03:04:48.4838743Z 2022-11-23T03:04:48.4838846Z OK (skipped=1) 2022-11-23T03:04:48.4838914Z 2022-11-23T03:04:48.4839038Z Generating XML reports... 2022-11-23T03:04:48.4839483Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025452.xml 2022-11-23T03:04:48.4839805Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4840191Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4840364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4840763Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4840944Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4840949Z 2022-11-23T03:04:48.4841046Z Running tests... 2022-11-23T03:04:48.4841324Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4842298Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82385 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.584s) 2022-11-23T03:04:48.4842304Z 2022-11-23T03:04:48.4842579Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4842686Z Ran 1 test in 0.584s 2022-11-23T03:04:48.4842692Z 2022-11-23T03:04:48.4842783Z OK (skipped=1) 2022-11-23T03:04:48.4842788Z 2022-11-23T03:04:48.4842910Z Generating XML reports... 2022-11-23T03:04:48.4843367Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025456.xml 2022-11-23T03:04:48.4843692Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4844082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4844255Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4844652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4844836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4844841Z 2022-11-23T03:04:48.4844945Z Running tests... 2022-11-23T03:04:48.4845211Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4846275Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82979 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.648s) 2022-11-23T03:04:48.4846287Z 2022-11-23T03:04:48.4846572Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4846683Z Ran 1 test in 0.648s 2022-11-23T03:04:48.4846689Z 2022-11-23T03:04:48.4846792Z OK (skipped=1) 2022-11-23T03:04:48.4846797Z 2022-11-23T03:04:48.4846917Z Generating XML reports... 2022-11-23T03:04:48.4847380Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025501.xml 2022-11-23T03:04:48.4847707Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4848093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4848245Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4848648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4848891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4848897Z 2022-11-23T03:04:48.4848999Z Running tests... 2022-11-23T03:04:48.4849280Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4850280Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82400 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.579s) 2022-11-23T03:04:48.4850286Z 2022-11-23T03:04:48.4850681Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4850789Z Ran 1 test in 0.579s 2022-11-23T03:04:48.4850795Z 2022-11-23T03:04:48.4850904Z OK (skipped=1) 2022-11-23T03:04:48.4850909Z 2022-11-23T03:04:48.4851033Z Generating XML reports... 2022-11-23T03:04:48.4851501Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025505.xml 2022-11-23T03:04:48.4851831Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4852217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4852377Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4852777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4852964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4852970Z 2022-11-23T03:04:48.4853072Z Running tests... 2022-11-23T03:04:48.4853348Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4854351Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82500 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.592s) 2022-11-23T03:04:48.4854357Z 2022-11-23T03:04:48.4854635Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4854748Z Ran 1 test in 0.593s 2022-11-23T03:04:48.4854753Z 2022-11-23T03:04:48.4854858Z OK (skipped=1) 2022-11-23T03:04:48.4854863Z 2022-11-23T03:04:48.4854971Z Generating XML reports... 2022-11-23T03:04:48.4855429Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025510.xml 2022-11-23T03:04:48.4855756Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4856217Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4856397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4856799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4856990Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4856996Z 2022-11-23T03:04:48.4857102Z Running tests... 2022-11-23T03:04:48.4857357Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4857656Z test_fp16 (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89911 2022-11-23T03:04:48.4857878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89912 2022-11-23T03:04:48.4858273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4858504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4858906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4859097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4859327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4859709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4859883Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4860278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4860472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4860706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4860952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69_ohgk1 2022-11-23T03:04:48.4861215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69_ohgk1/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4861469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n07azaw 2022-11-23T03:04:48.4861729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n07azaw/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4862025Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4862317Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4862419Z ok (7.118s) 2022-11-23T03:04:48.4862425Z 2022-11-23T03:04:48.4862707Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4862798Z Ran 1 test in 7.119s 2022-11-23T03:04:48.4862806Z 2022-11-23T03:04:48.4862901Z OK 2022-11-23T03:04:48.4862907Z 2022-11-23T03:04:48.4863031Z Generating XML reports... 2022-11-23T03:04:48.4863484Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025515.xml 2022-11-23T03:04:48.4863816Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4864207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4864388Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4864793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4864986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4864991Z 2022-11-23T03:04:48.4865105Z Running tests... 2022-11-23T03:04:48.4865438Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4865786Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90133 2022-11-23T03:04:48.4866008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90134 2022-11-23T03:04:48.4866408Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4866587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4866988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4867179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4867411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4868013Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4868412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4868587Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4868978Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4869163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4869381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4869922Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4870172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6dzkqjr 2022-11-23T03:04:48.4870431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6dzkqjr/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4870656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ww6yuei 2022-11-23T03:04:48.4870916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ww6yuei/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4871207Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4871491Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4871592Z ok (5.216s) 2022-11-23T03:04:48.4871600Z 2022-11-23T03:04:48.4871875Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4871981Z Ran 1 test in 5.217s 2022-11-23T03:04:48.4871987Z 2022-11-23T03:04:48.4872061Z OK 2022-11-23T03:04:48.4872067Z 2022-11-23T03:04:48.4872188Z Generating XML reports... 2022-11-23T03:04:48.4872651Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025526.xml 2022-11-23T03:04:48.4872975Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4873359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4873530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4873927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4874175Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4874183Z 2022-11-23T03:04:48.4874291Z Running tests... 2022-11-23T03:04:48.4874577Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4874912Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90353 2022-11-23T03:04:48.4875128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90354 2022-11-23T03:04:48.4875516Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4875686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4876061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4876248Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4876521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4877061Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4877452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4877625Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4878018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4878204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4878437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4878971Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4879216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39fa1i4v 2022-11-23T03:04:48.4879471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39fa1i4v/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4879716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfe9pgtfl 2022-11-23T03:04:48.4879977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfe9pgtfl/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4880273Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4880559Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4880652Z ok (5.215s) 2022-11-23T03:04:48.4880659Z 2022-11-23T03:04:48.4880935Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4881067Z Ran 1 test in 5.216s 2022-11-23T03:04:48.4881074Z 2022-11-23T03:04:48.4881179Z OK 2022-11-23T03:04:48.4881186Z 2022-11-23T03:04:48.4881328Z Generating XML reports... 2022-11-23T03:04:48.4881888Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025535.xml 2022-11-23T03:04:48.4882284Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4882742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4882949Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4883498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4883703Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4883738Z 2022-11-23T03:04:48.4883837Z Running tests... 2022-11-23T03:04:48.4884172Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4884555Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90573 2022-11-23T03:04:48.4884823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90574 2022-11-23T03:04:48.4885288Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4885500Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4885966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4886251Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4886529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4886992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4887203Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4887678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4887898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4888169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4888466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkuy1ury7 2022-11-23T03:04:48.4888794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkuy1ury7/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4889084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevntuc6_ 2022-11-23T03:04:48.4889396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevntuc6_/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4889744Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4890091Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4890185Z ok (7.166s) 2022-11-23T03:04:48.4890202Z 2022-11-23T03:04:48.4890508Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4890712Z Ran 1 test in 7.166s 2022-11-23T03:04:48.4890719Z 2022-11-23T03:04:48.4890823Z OK 2022-11-23T03:04:48.4890830Z 2022-11-23T03:04:48.4890973Z Generating XML reports... 2022-11-23T03:04:48.4891538Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025544.xml 2022-11-23T03:04:48.4891938Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4892399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4892610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4893089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4893319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4893326Z 2022-11-23T03:04:48.4893451Z Running tests... 2022-11-23T03:04:48.4893794Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4894215Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90795 2022-11-23T03:04:48.4894443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90796 2022-11-23T03:04:48.4894843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4895021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4895416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4895594Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4895822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4896215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4896393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4896830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4897026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4897261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4897522Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T03:04:48.4897776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49hh05ug 2022-11-23T03:04:48.4898035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49hh05ug/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4898305Z [W Module.cpp:969] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator()) 2022-11-23T03:04:48.4898556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmr8egt8 2022-11-23T03:04:48.4898826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmr8egt8/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4899122Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4899412Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4899646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4899879Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4900113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4900348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4900577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4900809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4901046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4901274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4901504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4901734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4901939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4902170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4902403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4902635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4902916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4903156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4903390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4903624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4903841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4904075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4904304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4904535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4904764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4905036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4905259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4905472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4905696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4905924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4906149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4906374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4906603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4906808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4907041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4907267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4907495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4907716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4907812Z ok (8.715s) 2022-11-23T03:04:48.4907818Z 2022-11-23T03:04:48.4908108Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4908215Z Ran 1 test in 8.715s 2022-11-23T03:04:48.4908221Z 2022-11-23T03:04:48.4908309Z OK 2022-11-23T03:04:48.4908315Z 2022-11-23T03:04:48.4908435Z Generating XML reports... 2022-11-23T03:04:48.4908898Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025555.xml 2022-11-23T03:04:48.4909222Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4909604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4909778Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4910176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4910367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4910373Z 2022-11-23T03:04:48.4910472Z Running tests... 2022-11-23T03:04:48.4910745Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4911074Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91017 2022-11-23T03:04:48.4911293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91018 2022-11-23T03:04:48.4911751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4911905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4912306Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4912490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4912710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4913096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4913267Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4913661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4913903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4914123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4914270Z skip: Need at least 4 CUDA devices (4.734s) 2022-11-23T03:04:48.4914276Z 2022-11-23T03:04:48.4914557Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4914665Z Ran 1 test in 4.734s 2022-11-23T03:04:48.4914671Z 2022-11-23T03:04:48.4914774Z OK (skipped=1) 2022-11-23T03:04:48.4914780Z 2022-11-23T03:04:48.4914900Z Generating XML reports... 2022-11-23T03:04:48.4915362Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025608.xml 2022-11-23T03:04:48.4915686Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4916070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4916249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4916647Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4916841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4916847Z 2022-11-23T03:04:48.4916951Z Running tests... 2022-11-23T03:04:48.4917207Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4917535Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91218 2022-11-23T03:04:48.4917754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91219 2022-11-23T03:04:48.4918139Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4918313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4918717Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4918903Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4919130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4919664Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4920242Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4920782Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4921315Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4921873Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4922445Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4922840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4923012Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4923409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4923576Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4923808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4924341Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4924873Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4925424Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4925949Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4926478Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4927053Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.4927158Z ok (4.615s) 2022-11-23T03:04:48.4927164Z 2022-11-23T03:04:48.4927449Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4927545Z Ran 1 test in 4.616s 2022-11-23T03:04:48.4927551Z 2022-11-23T03:04:48.4927642Z OK 2022-11-23T03:04:48.4927648Z 2022-11-23T03:04:48.4927768Z Generating XML reports... 2022-11-23T03:04:48.4928234Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025617.xml 2022-11-23T03:04:48.4928559Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4928928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4929149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4929552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4929744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4929750Z 2022-11-23T03:04:48.4929842Z Running tests... 2022-11-23T03:04:48.4930121Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4930464Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91419 2022-11-23T03:04:48.4930737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91420 2022-11-23T03:04:48.4931116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4931292Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4931689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4931879Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4932106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4932489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4932658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4933055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4933242Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4933475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4933725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_e99vlk3 2022-11-23T03:04:48.4933979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_e99vlk3/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4934211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5srllgik 2022-11-23T03:04:48.4934470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5srllgik/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4934740Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4935020Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4935113Z ok (7.218s) 2022-11-23T03:04:48.4935118Z 2022-11-23T03:04:48.4935394Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4935500Z Ran 1 test in 7.219s 2022-11-23T03:04:48.4935509Z 2022-11-23T03:04:48.4935597Z OK 2022-11-23T03:04:48.4935669Z 2022-11-23T03:04:48.4935795Z Generating XML reports... 2022-11-23T03:04:48.4936260Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025625.xml 2022-11-23T03:04:48.4936578Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4936965Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4937140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4937541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4937730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4937736Z 2022-11-23T03:04:48.4937839Z Running tests... 2022-11-23T03:04:48.4938115Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4938541Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91641 2022-11-23T03:04:48.4938759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91642 2022-11-23T03:04:48.4939153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4939322Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4939714Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4939891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4940096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4940482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4940658Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4941053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4941233Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4941455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4941705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyysmuf50 2022-11-23T03:04:48.4941965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyysmuf50/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4942208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe872tykm 2022-11-23T03:04:48.4942468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe872tykm/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4942765Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4943041Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4943143Z ok (7.112s) 2022-11-23T03:04:48.4943149Z 2022-11-23T03:04:48.4943432Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4943540Z Ran 1 test in 7.112s 2022-11-23T03:04:48.4943546Z 2022-11-23T03:04:48.4943641Z OK 2022-11-23T03:04:48.4943647Z 2022-11-23T03:04:48.4943769Z Generating XML reports... 2022-11-23T03:04:48.4944233Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025637.xml 2022-11-23T03:04:48.4944562Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4944956Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4945177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4945559Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4945757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4945762Z 2022-11-23T03:04:48.4945872Z Running tests... 2022-11-23T03:04:48.4946157Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4946523Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91863 2022-11-23T03:04:48.4946750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91864 2022-11-23T03:04:48.4947144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4947374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4947788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4947984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4948210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4948604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4948781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4949185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4949364Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4949596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4949852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_k2mxfol 2022-11-23T03:04:48.4950118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_k2mxfol/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4950366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf6q_pou 2022-11-23T03:04:48.4950617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf6q_pou/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4950897Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4951172Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4951381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4951603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4951692Z ok (7.221s) 2022-11-23T03:04:48.4951701Z 2022-11-23T03:04:48.4951970Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4952069Z Ran 1 test in 7.222s 2022-11-23T03:04:48.4952075Z 2022-11-23T03:04:48.4952154Z OK 2022-11-23T03:04:48.4952159Z 2022-11-23T03:04:48.4952271Z Generating XML reports... 2022-11-23T03:04:48.4952722Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025648.xml 2022-11-23T03:04:48.4953036Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4953412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4953575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4953962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4954141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4954151Z 2022-11-23T03:04:48.4954300Z Running tests... 2022-11-23T03:04:48.4954575Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4954934Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92085 2022-11-23T03:04:48.4955142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92086 2022-11-23T03:04:48.4955520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4955684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4956068Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4956243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4956499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4956880Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4957044Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4957430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4957608Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4957829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4958068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mj0e1gb 2022-11-23T03:04:48.4958317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mj0e1gb/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4958557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjt55iu3 2022-11-23T03:04:48.4958810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjt55iu3/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4959091Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4959370Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4959598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4959817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4959905Z ok (7.215s) 2022-11-23T03:04:48.4959911Z 2022-11-23T03:04:48.4960176Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4960279Z Ran 1 test in 7.215s 2022-11-23T03:04:48.4960284Z 2022-11-23T03:04:48.4960366Z OK 2022-11-23T03:04:48.4960371Z 2022-11-23T03:04:48.4960481Z Generating XML reports... 2022-11-23T03:04:48.4960940Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025659.xml 2022-11-23T03:04:48.4961254Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4961618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4961784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4962169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4962349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4962355Z 2022-11-23T03:04:48.4962450Z Running tests... 2022-11-23T03:04:48.4962719Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4963092Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92307 2022-11-23T03:04:48.4963306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92308 2022-11-23T03:04:48.4963685Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4963847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4964232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4964408Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4964627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4964998Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4965160Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4965605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4965783Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4966003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4966142Z skip: Need at least 4 CUDA devices (4.663s) 2022-11-23T03:04:48.4966149Z 2022-11-23T03:04:48.4966414Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4966515Z Ran 1 test in 4.663s 2022-11-23T03:04:48.4966520Z 2022-11-23T03:04:48.4966614Z OK (skipped=1) 2022-11-23T03:04:48.4966619Z 2022-11-23T03:04:48.4966719Z Generating XML reports... 2022-11-23T03:04:48.4967171Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025710.xml 2022-11-23T03:04:48.4967488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4967867Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4968028Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4968412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4968589Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4968595Z 2022-11-23T03:04:48.4968688Z Running tests... 2022-11-23T03:04:48.4968953Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4969271Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92508 2022-11-23T03:04:48.4969480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92509 2022-11-23T03:04:48.4969859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4970021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4970405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4970700Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4970919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4971302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4971461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4971844Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4972020Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4972304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4972441Z skip: Need at least 8 CUDA devices (4.613s) 2022-11-23T03:04:48.4972447Z 2022-11-23T03:04:48.4972706Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4972806Z Ran 1 test in 4.613s 2022-11-23T03:04:48.4972812Z 2022-11-23T03:04:48.4972903Z OK (skipped=1) 2022-11-23T03:04:48.4972908Z 2022-11-23T03:04:48.4973019Z Generating XML reports... 2022-11-23T03:04:48.4973471Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025719.xml 2022-11-23T03:04:48.4973785Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4974157Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4974387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4974787Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4974964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4974969Z 2022-11-23T03:04:48.4975063Z Running tests... 2022-11-23T03:04:48.4975326Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4975670Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92709 2022-11-23T03:04:48.4975874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92710 2022-11-23T03:04:48.4976245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4976409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4976799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4976975Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4977191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4977567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4977728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4978113Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4978279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4978490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4978580Z ok (5.203s) 2022-11-23T03:04:48.4978586Z 2022-11-23T03:04:48.4978854Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4978951Z Ran 1 test in 5.204s 2022-11-23T03:04:48.4978956Z 2022-11-23T03:04:48.4979033Z OK 2022-11-23T03:04:48.4979038Z 2022-11-23T03:04:48.4979147Z Generating XML reports... 2022-11-23T03:04:48.4979598Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025727.xml 2022-11-23T03:04:48.4979911Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4980284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4980446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4980834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4981065Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4981072Z 2022-11-23T03:04:48.4981168Z Running tests... 2022-11-23T03:04:48.4981438Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4981787Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92914 2022-11-23T03:04:48.4981996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92915 2022-11-23T03:04:48.4982372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4982536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4982923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4983100Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4983368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4983738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4983902Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4984286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4984464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4984681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4984814Z skip: Need at least 4 CUDA devices (5.226s) 2022-11-23T03:04:48.4984820Z 2022-11-23T03:04:48.4985085Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4985181Z Ran 1 test in 5.227s 2022-11-23T03:04:48.4985189Z 2022-11-23T03:04:48.4985284Z OK (skipped=1) 2022-11-23T03:04:48.4985291Z 2022-11-23T03:04:48.4985403Z Generating XML reports... 2022-11-23T03:04:48.4985852Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025737.xml 2022-11-23T03:04:48.4986165Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4986538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4986699Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4987088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4987263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4987269Z 2022-11-23T03:04:48.4987361Z Running tests... 2022-11-23T03:04:48.4987626Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4987984Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93115 2022-11-23T03:04:48.4988193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93116 2022-11-23T03:04:48.4988565Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4988724Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4989097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4989274Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4989488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4989908Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4990076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4990459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4990640Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4990852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4991089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvuwntxg 2022-11-23T03:04:48.4991339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvuwntxg/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4991577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9bu2kq0 2022-11-23T03:04:48.4991824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9bu2kq0/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4992212Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4992488Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.4992707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4992924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.4993010Z ok (7.118s) 2022-11-23T03:04:48.4993016Z 2022-11-23T03:04:48.4993280Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4993376Z Ran 1 test in 7.118s 2022-11-23T03:04:48.4993382Z 2022-11-23T03:04:48.4993458Z OK 2022-11-23T03:04:48.4993463Z 2022-11-23T03:04:48.4993571Z Generating XML reports... 2022-11-23T03:04:48.4994010Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025746.xml 2022-11-23T03:04:48.4994330Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.4994703Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4994864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4995247Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4995423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4995429Z 2022-11-23T03:04:48.4995520Z Running tests... 2022-11-23T03:04:48.4995785Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.4996137Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93337 2022-11-23T03:04:48.4996342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93338 2022-11-23T03:04:48.4996729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4996889Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4997273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4997448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4997667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.4998036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.4998195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.4998577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.4998805Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.4999022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.4999259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewrx5y8j 2022-11-23T03:04:48.4999506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewrx5y8j/_remote_module_non_scriptable.py 2022-11-23T03:04:48.4999730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xrnv9g_ 2022-11-23T03:04:48.4999975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xrnv9g_/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5000255Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5000528Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5000748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.5001012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.5001097Z ok (7.424s) 2022-11-23T03:04:48.5001103Z 2022-11-23T03:04:48.5001374Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5001469Z Ran 1 test in 7.424s 2022-11-23T03:04:48.5001475Z 2022-11-23T03:04:48.5001551Z OK 2022-11-23T03:04:48.5001556Z 2022-11-23T03:04:48.5001663Z Generating XML reports... 2022-11-23T03:04:48.5002112Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025757.xml 2022-11-23T03:04:48.5002426Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5002799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5002960Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5003356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5003533Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5003539Z 2022-11-23T03:04:48.5003631Z Running tests... 2022-11-23T03:04:48.5003896Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5004222Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93559 2022-11-23T03:04:48.5004429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93560 2022-11-23T03:04:48.5004794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5004955Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5005339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5005521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5005735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5006167Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5006360Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5006825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5007036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5007287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5007626Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5008020Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5008123Z ok (20.247s) 2022-11-23T03:04:48.5008131Z 2022-11-23T03:04:48.5008454Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5008569Z Ran 1 test in 20.248s 2022-11-23T03:04:48.5008576Z 2022-11-23T03:04:48.5008664Z OK 2022-11-23T03:04:48.5008671Z 2022-11-23T03:04:48.5008804Z Generating XML reports... 2022-11-23T03:04:48.5009346Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025808.xml 2022-11-23T03:04:48.5009724Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5010174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5010366Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5010963Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5011180Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5011186Z 2022-11-23T03:04:48.5011301Z Running tests... 2022-11-23T03:04:48.5011616Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5011807Z test_no_grad (__main__.DistributedDataParallelTest) 2022-11-23T03:04:48.5012163Z Note: this test can be sped up by only running it on a CPU module ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93775 2022-11-23T03:04:48.5012407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93776 2022-11-23T03:04:48.5012857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5013050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5013518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5013727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5013986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5014439Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5014628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5015094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5015310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5015570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5015850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2z73jve 2022-11-23T03:04:48.5016153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2z73jve/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5016437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgcqlnz0w 2022-11-23T03:04:48.5016734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgcqlnz0w/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5017056Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5017385Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5017486Z ok (7.326s) 2022-11-23T03:04:48.5017493Z 2022-11-23T03:04:48.5017812Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5017928Z Ran 1 test in 7.327s 2022-11-23T03:04:48.5017934Z 2022-11-23T03:04:48.5018023Z OK 2022-11-23T03:04:48.5018030Z 2022-11-23T03:04:48.5018161Z Generating XML reports... 2022-11-23T03:04:48.5018787Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025832.xml 2022-11-23T03:04:48.5019172Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5019625Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5019821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5020284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5020496Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5020503Z 2022-11-23T03:04:48.5020610Z Running tests... 2022-11-23T03:04:48.5020925Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5021316Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93989 2022-11-23T03:04:48.5021625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93990 2022-11-23T03:04:48.5022076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5022266Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5022731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5022942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5023189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5023638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5023838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5024229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5024405Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5024619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5024853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rf2t6fp 2022-11-23T03:04:48.5025105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rf2t6fp/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5025336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpow369r28 2022-11-23T03:04:48.5025579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpow369r28/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5025854Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5026133Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5026221Z ok (5.425s) 2022-11-23T03:04:48.5026227Z 2022-11-23T03:04:48.5026490Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5026585Z Ran 1 test in 5.425s 2022-11-23T03:04:48.5026591Z 2022-11-23T03:04:48.5026669Z OK 2022-11-23T03:04:48.5026675Z 2022-11-23T03:04:48.5026782Z Generating XML reports... 2022-11-23T03:04:48.5027231Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025844.xml 2022-11-23T03:04:48.5027545Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5027919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5028081Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5028506Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5028689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5028695Z 2022-11-23T03:04:48.5028788Z Running tests... 2022-11-23T03:04:48.5029055Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5029364Z test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94201 2022-11-23T03:04:48.5029570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94202 2022-11-23T03:04:48.5029947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5030107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5030494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5030722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5030938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5031167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5031543Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5031705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5032090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5032266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5032482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5032707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5033113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5033515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5033600Z ok (4.615s) 2022-11-23T03:04:48.5033605Z 2022-11-23T03:04:48.5033868Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5033955Z Ran 1 test in 4.615s 2022-11-23T03:04:48.5033969Z 2022-11-23T03:04:48.5034037Z OK 2022-11-23T03:04:48.5034042Z 2022-11-23T03:04:48.5034150Z Generating XML reports... 2022-11-23T03:04:48.5034598Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025853.xml 2022-11-23T03:04:48.5034909Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5035289Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5035452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5035836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5036011Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5036016Z 2022-11-23T03:04:48.5036108Z Running tests... 2022-11-23T03:04:48.5036373Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5036693Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94406 2022-11-23T03:04:48.5036898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94407 2022-11-23T03:04:48.5037325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5037493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5037883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5038059Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5038275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5038809Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5039188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5039398Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5039785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5039962Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5040177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5040697Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5040933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqk55tb5p 2022-11-23T03:04:48.5041184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqk55tb5p/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5041412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyke4hd3 2022-11-23T03:04:48.5041665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyke4hd3/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5041943Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5042216Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5042734Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5043249Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5043787Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5044310Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5044916Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5045437Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5045954Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5046509Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5047031Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5047545Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5048064Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5048590Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5049115Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5049635Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5049720Z ok (5.220s) 2022-11-23T03:04:48.5049726Z 2022-11-23T03:04:48.5050002Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5050099Z Ran 1 test in 5.220s 2022-11-23T03:04:48.5050105Z 2022-11-23T03:04:48.5050180Z OK 2022-11-23T03:04:48.5050186Z 2022-11-23T03:04:48.5050294Z Generating XML reports... 2022-11-23T03:04:48.5050860Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025902.xml 2022-11-23T03:04:48.5051187Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5051563Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5051725Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5052113Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5052288Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5052294Z 2022-11-23T03:04:48.5052389Z Running tests... 2022-11-23T03:04:48.5052652Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5052991Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94626 2022-11-23T03:04:48.5053245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94627 2022-11-23T03:04:48.5053623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5053784Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5054172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5054347Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5054554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5055087Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5055467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5055618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5056002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5056178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5056396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5056919Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5057155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lf27rde 2022-11-23T03:04:48.5057402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lf27rde/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5057636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pk41a34 2022-11-23T03:04:48.5057881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pk41a34/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5058158Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5058431Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5058999Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5059519Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5060038Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5060550Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5061106Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5061627Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5062145Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5062664Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5063183Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5063699Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5064216Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5064769Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True 2022-11-23T03:04:48.5065292Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5065807Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False 2022-11-23T03:04:48.5065893Z ok (5.331s) 2022-11-23T03:04:48.5065899Z 2022-11-23T03:04:48.5066174Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5066315Z Ran 1 test in 5.331s 2022-11-23T03:04:48.5066321Z 2022-11-23T03:04:48.5066397Z OK 2022-11-23T03:04:48.5066402Z 2022-11-23T03:04:48.5066510Z Generating XML reports... 2022-11-23T03:04:48.5066967Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025911.xml 2022-11-23T03:04:48.5067282Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5067654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5067814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5068202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5068379Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5068388Z 2022-11-23T03:04:48.5068480Z Running tests... 2022-11-23T03:04:48.5068767Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5069084Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94846 2022-11-23T03:04:48.5069282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94847 2022-11-23T03:04:48.5069657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5069819Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5070204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5070378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5070593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5070973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5071135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5071519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5071695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5071910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5072143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_prq5saf 2022-11-23T03:04:48.5072390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_prq5saf/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5072628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2mcztir 2022-11-23T03:04:48.5072931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2mcztir/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5073216Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5073490Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5073713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.5073935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.5074021Z ok (8.125s) 2022-11-23T03:04:48.5074027Z 2022-11-23T03:04:48.5074291Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5074379Z Ran 1 test in 8.126s 2022-11-23T03:04:48.5074396Z 2022-11-23T03:04:48.5074463Z OK 2022-11-23T03:04:48.5074469Z 2022-11-23T03:04:48.5074577Z Generating XML reports... 2022-11-23T03:04:48.5075027Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025921.xml 2022-11-23T03:04:48.5075392Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5075768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5075928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5076314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5076489Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5076495Z 2022-11-23T03:04:48.5076586Z Running tests... 2022-11-23T03:04:48.5076852Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5077178Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95107 2022-11-23T03:04:48.5077389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95108 2022-11-23T03:04:48.5077766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5077930Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5078317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5078493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5078707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5079082Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5079246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5079639Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5079819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5080024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5080259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelvk53jf 2022-11-23T03:04:48.5080508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelvk53jf/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5080745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2i08vvh_ 2022-11-23T03:04:48.5080994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2i08vvh_/_remote_module_non_scriptable.py 2022-11-23T03:04:48.5081275Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5081548Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5081819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.5082038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:04:48.5082127Z ok (5.524s) 2022-11-23T03:04:48.5082133Z 2022-11-23T03:04:48.5082398Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5082496Z Ran 1 test in 5.524s 2022-11-23T03:04:48.5082502Z 2022-11-23T03:04:48.5082578Z OK 2022-11-23T03:04:48.5082584Z 2022-11-23T03:04:48.5082695Z Generating XML reports... 2022-11-23T03:04:48.5083145Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123025933.xml 2022-11-23T03:04:48.5083461Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5083836Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5084051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5084440Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5084616Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5084622Z 2022-11-23T03:04:48.5084716Z Running tests... 2022-11-23T03:04:48.5084971Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5085283Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95327 2022-11-23T03:04:48.5085491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95328 2022-11-23T03:04:48.5085696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95329 2022-11-23T03:04:48.5086070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5086237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5086626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5086802Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5087018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5087390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5087550Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5087936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5088113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5088328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:04:48.5088705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5088868Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5089254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5089428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5089641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5089777Z skip: Need at least 3 CUDA devices (5.234s) 2022-11-23T03:04:48.5089783Z 2022-11-23T03:04:48.5090052Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5090139Z Ran 1 test in 5.234s 2022-11-23T03:04:48.5090155Z 2022-11-23T03:04:48.5090238Z OK (skipped=1) 2022-11-23T03:04:48.5090246Z 2022-11-23T03:04:48.5090355Z Generating XML reports... 2022-11-23T03:04:48.5091013Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025943.xml 2022-11-23T03:04:48.5091343Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5091718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5091884Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5092269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5092448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5092454Z 2022-11-23T03:04:48.5092545Z Running tests... 2022-11-23T03:04:48.5092809Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5093177Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95595 2022-11-23T03:04:48.5093387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95596 2022-11-23T03:04:48.5093592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95597 2022-11-23T03:04:48.5093970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5094133Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5094519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5094696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5094909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:04:48.5095287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5095452Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5095834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5096000Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5096217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5096590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5096754Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5097141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5097316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5097538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5097672Z skip: Need at least 3 CUDA devices (5.227s) 2022-11-23T03:04:48.5097678Z 2022-11-23T03:04:48.5097942Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5098037Z Ran 1 test in 5.227s 2022-11-23T03:04:48.5098043Z 2022-11-23T03:04:48.5098134Z OK (skipped=1) 2022-11-23T03:04:48.5098139Z 2022-11-23T03:04:48.5098252Z Generating XML reports... 2022-11-23T03:04:48.5098679Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123025952.xml 2022-11-23T03:04:48.5098992Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5099364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5099526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5099966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5100146Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5100152Z 2022-11-23T03:04:48.5100243Z Running tests... 2022-11-23T03:04:48.5100512Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5100826Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: Frequently times out see https://github.com/pytorch/pytorch/issues/58920 (0.000s) 2022-11-23T03:04:48.5100831Z 2022-11-23T03:04:48.5101094Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5101181Z Ran 1 test in 0.001s 2022-11-23T03:04:48.5101194Z 2022-11-23T03:04:48.5101277Z OK (skipped=1) 2022-11-23T03:04:48.5101283Z 2022-11-23T03:04:48.5101393Z Generating XML reports... 2022-11-23T03:04:48.5101889Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123030001.xml 2022-11-23T03:04:48.5102202Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5102576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5102740Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5103122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5103298Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5103304Z 2022-11-23T03:04:48.5103398Z Running tests... 2022-11-23T03:04:48.5103665Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5103980Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95929 2022-11-23T03:04:48.5104196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95930 2022-11-23T03:04:48.5104401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95931 2022-11-23T03:04:48.5104777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5104938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5105321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5105500Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5105713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5106085Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5106250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5106635Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5106801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5107020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:04:48.5107392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5107552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5107936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5108110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5108323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5108510Z skip: Need at least 3 CUDA devices (4.729s) 2022-11-23T03:04:48.5108517Z 2022-11-23T03:04:48.5108790Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5108889Z Ran 1 test in 4.729s 2022-11-23T03:04:48.5108895Z 2022-11-23T03:04:48.5108988Z OK (skipped=1) 2022-11-23T03:04:48.5108994Z 2022-11-23T03:04:48.5109105Z Generating XML reports... 2022-11-23T03:04:48.5109529Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123030005.xml 2022-11-23T03:04:48.5109843Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5110214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5110376Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5110764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5110987Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5110992Z 2022-11-23T03:04:48.5111086Z Running tests... 2022-11-23T03:04:48.5111352Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5111666Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96197 2022-11-23T03:04:48.5111873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96198 2022-11-23T03:04:48.5112070Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96199 2022-11-23T03:04:48.5112442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5112602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5112993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5113170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5113384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5113754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5113918Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5114304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5114482Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5114699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5115073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5115236Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5115620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5115794Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5116011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:04:48.5116096Z ok (5.454s) 2022-11-23T03:04:48.5116102Z 2022-11-23T03:04:48.5116366Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5116462Z Ran 1 test in 5.454s 2022-11-23T03:04:48.5116469Z 2022-11-23T03:04:48.5116547Z OK 2022-11-23T03:04:48.5116552Z 2022-11-23T03:04:48.5116662Z Generating XML reports... 2022-11-23T03:04:48.5117079Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123030014.xml 2022-11-23T03:04:48.5117446Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5117826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5117986Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5118374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5118552Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5118558Z 2022-11-23T03:04:48.5118649Z Running tests... 2022-11-23T03:04:48.5118919Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5119227Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96465 2022-11-23T03:04:48.5119481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96466 2022-11-23T03:04:48.5119686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96467 2022-11-23T03:04:48.5120065Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5120226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5120609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5120789Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5121002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5121374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5121536Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5121925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5122101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5122316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5122686Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5122839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5123222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5123397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5123609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:04:48.5123697Z ok (4.743s) 2022-11-23T03:04:48.5123703Z 2022-11-23T03:04:48.5123970Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5124068Z Ran 1 test in 4.743s 2022-11-23T03:04:48.5124073Z 2022-11-23T03:04:48.5124152Z OK 2022-11-23T03:04:48.5124157Z 2022-11-23T03:04:48.5124268Z Generating XML reports... 2022-11-23T03:04:48.5124694Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123030024.xml 2022-11-23T03:04:48.5125005Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5125379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5125540Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5125923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5126152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5126158Z 2022-11-23T03:04:48.5126254Z Running tests... 2022-11-23T03:04:48.5126521Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5126831Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96733 2022-11-23T03:04:48.5127036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96734 2022-11-23T03:04:48.5127244Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96735 2022-11-23T03:04:48.5127619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5127772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5128155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5128386Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5128605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5128980Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5129140Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5129521Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5129698Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5129916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5130287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5130455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5130907Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5131081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5131299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:04:48.5131387Z ok (4.845s) 2022-11-23T03:04:48.5131393Z 2022-11-23T03:04:48.5131707Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5131820Z Ran 1 test in 4.845s 2022-11-23T03:04:48.5131827Z 2022-11-23T03:04:48.5131916Z OK 2022-11-23T03:04:48.5131923Z 2022-11-23T03:04:48.5132055Z Generating XML reports... 2022-11-23T03:04:48.5132569Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123030033.xml 2022-11-23T03:04:48.5132952Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5133396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5133592Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5134053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5134261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5134268Z 2022-11-23T03:04:48.5134384Z Running tests... 2022-11-23T03:04:48.5134699Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5134997Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: Test does not pass when run locally (0.001s) 2022-11-23T03:04:48.5135004Z 2022-11-23T03:04:48.5135317Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5135430Z Ran 1 test in 0.002s 2022-11-23T03:04:48.5135440Z 2022-11-23T03:04:48.5135626Z OK (skipped=1) 2022-11-23T03:04:48.5135634Z 2022-11-23T03:04:48.5135772Z Generating XML reports... 2022-11-23T03:04:48.5136290Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123030042.xml 2022-11-23T03:04:48.5136669Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5137116Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5137314Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5137776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5137982Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5137989Z 2022-11-23T03:04:48.5138103Z Running tests... 2022-11-23T03:04:48.5138487Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5138833Z test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97067 2022-11-23T03:04:48.5139087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97068 2022-11-23T03:04:48.5139333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97069 2022-11-23T03:04:48.5139778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5139967Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5140432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5140645Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5140903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5141355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5141548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5142005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5142216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5142473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5142923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5143113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5143578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5143795Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5144057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:04:48.5144223Z skip: Need at least 3 CUDA devices (4.949s) 2022-11-23T03:04:48.5144230Z 2022-11-23T03:04:48.5144501Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5144598Z Ran 1 test in 4.949s 2022-11-23T03:04:48.5144604Z 2022-11-23T03:04:48.5144695Z OK (skipped=1) 2022-11-23T03:04:48.5144701Z 2022-11-23T03:04:48.5144810Z Generating XML reports... 2022-11-23T03:04:48.5145235Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123030046.xml 2022-11-23T03:04:48.5145547Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5145909Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5146127Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5146515Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5146689Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5146694Z 2022-11-23T03:04:48.5146785Z Running tests... 2022-11-23T03:04:48.5147053Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5147415Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97335 2022-11-23T03:04:48.5147786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5147948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5148335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5148558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5148773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5148997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5149406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:04:48.5149680Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5149765Z ok (5.022s) 2022-11-23T03:04:48.5149771Z 2022-11-23T03:04:48.5150034Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5150130Z Ran 1 test in 5.023s 2022-11-23T03:04:48.5150135Z 2022-11-23T03:04:48.5150210Z OK 2022-11-23T03:04:48.5150219Z 2022-11-23T03:04:48.5150326Z Generating XML reports... 2022-11-23T03:04:48.5150862Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123030055.xml 2022-11-23T03:04:48.5151176Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5151539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5151701Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5152083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5152260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5152266Z 2022-11-23T03:04:48.5152358Z Running tests... 2022-11-23T03:04:48.5152622Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5152995Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97476 2022-11-23T03:04:48.5153370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5153532Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5153915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5154091Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5154306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5154530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5154930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:04:48.5155259Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5156025Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions 2022-11-23T03:04:48.5156118Z warnings.warn( 2022-11-23T03:04:48.5156203Z ok (5.134s) 2022-11-23T03:04:48.5156209Z 2022-11-23T03:04:48.5156471Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5156566Z Ran 1 test in 5.134s 2022-11-23T03:04:48.5156572Z 2022-11-23T03:04:48.5156649Z OK 2022-11-23T03:04:48.5156654Z 2022-11-23T03:04:48.5156762Z Generating XML reports... 2022-11-23T03:04:48.5157296Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123030104.xml 2022-11-23T03:04:48.5157650Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5158024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5158188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5158573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5158750Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5158755Z 2022-11-23T03:04:48.5158846Z Running tests... 2022-11-23T03:04:48.5159110Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5159469Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97615 2022-11-23T03:04:48.5159845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5160005Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5160388Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5160561Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5160773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5160998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5161400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:04:48.5161678Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5161763Z ok (5.008s) 2022-11-23T03:04:48.5161772Z 2022-11-23T03:04:48.5162036Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5162134Z Ran 1 test in 5.009s 2022-11-23T03:04:48.5162140Z 2022-11-23T03:04:48.5162215Z OK 2022-11-23T03:04:48.5162220Z 2022-11-23T03:04:48.5162328Z Generating XML reports... 2022-11-23T03:04:48.5162864Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123030113.xml 2022-11-23T03:04:48.5163168Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5163542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5163704Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5164086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5164315Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5164321Z 2022-11-23T03:04:48.5164416Z Running tests... 2022-11-23T03:04:48.5164685Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5165052Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97754 2022-11-23T03:04:48.5165424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5165584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5165969Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5166144Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5166361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5166641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5167047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:04:48.5167320Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5167406Z ok (4.928s) 2022-11-23T03:04:48.5167412Z 2022-11-23T03:04:48.5167676Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5167773Z Ran 1 test in 4.928s 2022-11-23T03:04:48.5167778Z 2022-11-23T03:04:48.5167854Z OK 2022-11-23T03:04:48.5167859Z 2022-11-23T03:04:48.5167967Z Generating XML reports... 2022-11-23T03:04:48.5168500Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123030122.xml 2022-11-23T03:04:48.5168813Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5169181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5169342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5169726Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5169901Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5169907Z 2022-11-23T03:04:48.5169998Z Running tests... 2022-11-23T03:04:48.5170260Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5170506Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: GPUs are available, skipping test (0.001s) 2022-11-23T03:04:48.5170513Z 2022-11-23T03:04:48.5170926Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5171023Z Ran 1 test in 0.001s 2022-11-23T03:04:48.5171034Z 2022-11-23T03:04:48.5171129Z OK (skipped=1) 2022-11-23T03:04:48.5171134Z 2022-11-23T03:04:48.5171244Z Generating XML reports... 2022-11-23T03:04:48.5171733Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221123030131.xml 2022-11-23T03:04:48.5172104Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5172548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5172739Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5173203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5173413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5173420Z 2022-11-23T03:04:48.5173526Z Running tests... 2022-11-23T03:04:48.5173948Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5174314Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97961 2022-11-23T03:04:48.5174562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97962 2022-11-23T03:04:48.5175015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5175200Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5175663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5175876Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5176129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5176400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5176914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5177110Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5177569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5177777Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5178039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5178309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5178796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5179273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5179379Z ok (5.018s) 2022-11-23T03:04:48.5179386Z 2022-11-23T03:04:48.5179700Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5179811Z Ran 1 test in 5.019s 2022-11-23T03:04:48.5179817Z 2022-11-23T03:04:48.5179913Z OK 2022-11-23T03:04:48.5179919Z 2022-11-23T03:04:48.5180046Z Generating XML reports... 2022-11-23T03:04:48.5180556Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030135.xml 2022-11-23T03:04:48.5180926Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5181377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5181574Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5182027Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5182247Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5182253Z 2022-11-23T03:04:48.5182369Z Running tests... 2022-11-23T03:04:48.5182694Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5183048Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98166 2022-11-23T03:04:48.5183297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98167 2022-11-23T03:04:48.5183749Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5183941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5184394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5184620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5184837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5185061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5185438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5185600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5185983Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5186158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5186374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5186599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5187057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5187452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5187727Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5188000Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5188076Z ok (5.236s) 2022-11-23T03:04:48.5188089Z 2022-11-23T03:04:48.5188343Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5188439Z Ran 1 test in 5.236s 2022-11-23T03:04:48.5188445Z 2022-11-23T03:04:48.5188521Z OK 2022-11-23T03:04:48.5188527Z 2022-11-23T03:04:48.5188635Z Generating XML reports... 2022-11-23T03:04:48.5189061Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030144.xml 2022-11-23T03:04:48.5189378Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5189751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5189914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5190296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5190471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5190477Z 2022-11-23T03:04:48.5190567Z Running tests... 2022-11-23T03:04:48.5190831Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5191122Z test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98378 2022-11-23T03:04:48.5191332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98379 2022-11-23T03:04:48.5191705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5191866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5192249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5192425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5192638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5192863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5193227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5193393Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5193830Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5194009Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5194225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5194446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5194854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5195250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5195524Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5195797Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5195931Z ok (5.126s) 2022-11-23T03:04:48.5195937Z 2022-11-23T03:04:48.5196203Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5196298Z Ran 1 test in 5.126s 2022-11-23T03:04:48.5196303Z 2022-11-23T03:04:48.5196380Z OK 2022-11-23T03:04:48.5196385Z 2022-11-23T03:04:48.5196492Z Generating XML reports... 2022-11-23T03:04:48.5196917Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030153.xml 2022-11-23T03:04:48.5197228Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5197599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5197760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5198146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5198328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5198334Z 2022-11-23T03:04:48.5198426Z Running tests... 2022-11-23T03:04:48.5198681Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5198970Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98594 2022-11-23T03:04:48.5199178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98595 2022-11-23T03:04:48.5199554Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5199714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5200097Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5200272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5200494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5200718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5201092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5201253Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5201636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5201811Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5202024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5202246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5202703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5203106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5203384Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5203657Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5203740Z ok (5.296s) 2022-11-23T03:04:48.5203746Z 2022-11-23T03:04:48.5204008Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5204094Z Ran 1 test in 5.296s 2022-11-23T03:04:48.5204110Z 2022-11-23T03:04:48.5204177Z OK 2022-11-23T03:04:48.5204182Z 2022-11-23T03:04:48.5204292Z Generating XML reports... 2022-11-23T03:04:48.5204718Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030202.xml 2022-11-23T03:04:48.5205083Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5205455Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5205615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5205999Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5206174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5206179Z 2022-11-23T03:04:48.5206271Z Running tests... 2022-11-23T03:04:48.5206535Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5206816Z test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98810 2022-11-23T03:04:48.5207023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98811 2022-11-23T03:04:48.5207401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5207561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5207944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5208119Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5208335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5208558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5208931Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5209092Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5209484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5209651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5209863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5210090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5210489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5210955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5211230Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5211504Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5211592Z ok (5.064s) 2022-11-23T03:04:48.5211598Z 2022-11-23T03:04:48.5211921Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5212018Z Ran 1 test in 5.064s 2022-11-23T03:04:48.5212023Z 2022-11-23T03:04:48.5212101Z OK 2022-11-23T03:04:48.5212107Z 2022-11-23T03:04:48.5212214Z Generating XML reports... 2022-11-23T03:04:48.5212644Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030211.xml 2022-11-23T03:04:48.5212956Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5213331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5213491Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5213873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5214122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5214128Z 2022-11-23T03:04:48.5214221Z Running tests... 2022-11-23T03:04:48.5214489Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5214776Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99022 2022-11-23T03:04:48.5214981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99023 2022-11-23T03:04:48.5215347Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5215510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5215895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5216071Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5216294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5216518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5216892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5217052Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5217434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5217611Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5217823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5218045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5218447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5218845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5219119Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5219393Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5219477Z ok (5.015s) 2022-11-23T03:04:48.5219483Z 2022-11-23T03:04:48.5219747Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5219843Z Ran 1 test in 5.016s 2022-11-23T03:04:48.5219849Z 2022-11-23T03:04:48.5219925Z OK 2022-11-23T03:04:48.5219930Z 2022-11-23T03:04:48.5220037Z Generating XML reports... 2022-11-23T03:04:48.5220453Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030220.xml 2022-11-23T03:04:48.5220821Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5221204Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5221364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5221748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5221923Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5221929Z 2022-11-23T03:04:48.5222020Z Running tests... 2022-11-23T03:04:48.5222284Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5222570Z test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99234 2022-11-23T03:04:48.5222779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99235 2022-11-23T03:04:48.5223209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5223369Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5223752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5223926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5224139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5224362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5224734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5224896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5225283Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5225462Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5225676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5225903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5226293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5226687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5226960Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5227232Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5227323Z ok (5.062s) 2022-11-23T03:04:48.5227329Z 2022-11-23T03:04:48.5227601Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5227697Z Ran 1 test in 5.062s 2022-11-23T03:04:48.5227703Z 2022-11-23T03:04:48.5227785Z OK 2022-11-23T03:04:48.5227791Z 2022-11-23T03:04:48.5227899Z Generating XML reports... 2022-11-23T03:04:48.5228323Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030229.xml 2022-11-23T03:04:48.5228633Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5229003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5229165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5229546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5229721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5229729Z 2022-11-23T03:04:48.5229871Z Running tests... 2022-11-23T03:04:48.5230143Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5230434Z test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99446 2022-11-23T03:04:48.5230642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99447 2022-11-23T03:04:48.5231014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5231175Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5231556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5231722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5231939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5232212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5232588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5232748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5233134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5233309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5233525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5233748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5234147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5234544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5234631Z ok (4.908s) 2022-11-23T03:04:48.5234637Z 2022-11-23T03:04:48.5234901Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5234995Z Ran 1 test in 4.908s 2022-11-23T03:04:48.5235001Z 2022-11-23T03:04:48.5235078Z OK 2022-11-23T03:04:48.5235083Z 2022-11-23T03:04:48.5235193Z Generating XML reports... 2022-11-23T03:04:48.5235615Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030238.xml 2022-11-23T03:04:48.5235927Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5236297Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5236457Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5236847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5237024Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5237030Z 2022-11-23T03:04:48.5237113Z Running tests... 2022-11-23T03:04:48.5237377Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5237664Z test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99651 2022-11-23T03:04:48.5237868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99652 2022-11-23T03:04:48.5238243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5238403Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5238842Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5239021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5239234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5239461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5239841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5240002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5240387Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5240564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5240780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5241068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5241473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5241867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5242142Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5242416Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5242500Z ok (5.015s) 2022-11-23T03:04:48.5242506Z 2022-11-23T03:04:48.5242758Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5242855Z Ran 1 test in 5.015s 2022-11-23T03:04:48.5242861Z 2022-11-23T03:04:48.5242936Z OK 2022-11-23T03:04:48.5242942Z 2022-11-23T03:04:48.5243053Z Generating XML reports... 2022-11-23T03:04:48.5243480Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030247.xml 2022-11-23T03:04:48.5243792Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5244164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5244325Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5244709Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5244886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5244891Z 2022-11-23T03:04:48.5244982Z Running tests... 2022-11-23T03:04:48.5245245Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5245535Z test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99863 2022-11-23T03:04:48.5245742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99864 2022-11-23T03:04:48.5246118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5246279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5246660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5246833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5247051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5247280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5247654Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5247870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5248251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5248430Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5248646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5248870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5249270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5249666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5249939Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5250264Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5250354Z ok (9.412s) 2022-11-23T03:04:48.5250359Z 2022-11-23T03:04:48.5250673Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5250769Z Ran 1 test in 9.413s 2022-11-23T03:04:48.5250775Z 2022-11-23T03:04:48.5250851Z OK 2022-11-23T03:04:48.5250856Z 2022-11-23T03:04:48.5250964Z Generating XML reports... 2022-11-23T03:04:48.5251388Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030256.xml 2022-11-23T03:04:48.5251699Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5252069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5252231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5252620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5252796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5252802Z 2022-11-23T03:04:48.5252893Z Running tests... 2022-11-23T03:04:48.5253157Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5253452Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100075 2022-11-23T03:04:48.5253658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100076 2022-11-23T03:04:48.5254031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5254192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5254578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5254757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5254972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5255199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5255572Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5255733Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5256117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5256292Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5256538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5256887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5257374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5257848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5258185Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5258517Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5258617Z ok (5.706s) 2022-11-23T03:04:48.5258624Z 2022-11-23T03:04:48.5258939Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5259054Z Ran 1 test in 5.706s 2022-11-23T03:04:48.5259061Z 2022-11-23T03:04:48.5259153Z OK 2022-11-23T03:04:48.5259160Z 2022-11-23T03:04:48.5259277Z Generating XML reports... 2022-11-23T03:04:48.5259859Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030309.xml 2022-11-23T03:04:48.5260239Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5260684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5260881Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5261346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5261554Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5261560Z 2022-11-23T03:04:48.5261674Z Running tests... 2022-11-23T03:04:48.5261990Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5262333Z test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100285 2022-11-23T03:04:48.5262591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100286 2022-11-23T03:04:48.5263044Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5263232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5263697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5263908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5264169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5264438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5264886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5265087Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5265546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5265758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5266017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5266280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5266766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5267244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5267580Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5267975Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5268076Z ok (5.413s) 2022-11-23T03:04:48.5268084Z 2022-11-23T03:04:48.5268406Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5268521Z Ran 1 test in 5.414s 2022-11-23T03:04:48.5268528Z 2022-11-23T03:04:48.5268617Z OK 2022-11-23T03:04:48.5268624Z 2022-11-23T03:04:48.5268754Z Generating XML reports... 2022-11-23T03:04:48.5269261Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030319.xml 2022-11-23T03:04:48.5269638Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5270086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5270279Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5270747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5271017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5271023Z 2022-11-23T03:04:48.5271131Z Running tests... 2022-11-23T03:04:48.5271448Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5271828Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100501 2022-11-23T03:04:48.5272072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100502 2022-11-23T03:04:48.5272523Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5272719Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5273170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5273393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5273656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5273925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5274381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5274562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5274949Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5275124Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5275336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5275566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5275965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5276358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5276444Z ok (5.012s) 2022-11-23T03:04:48.5276450Z 2022-11-23T03:04:48.5276712Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5276808Z Ran 1 test in 5.013s 2022-11-23T03:04:48.5276814Z 2022-11-23T03:04:48.5276890Z OK 2022-11-23T03:04:48.5276895Z 2022-11-23T03:04:48.5277004Z Generating XML reports... 2022-11-23T03:04:48.5277429Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030328.xml 2022-11-23T03:04:48.5277740Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5278232Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5278396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5278781Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5278948Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5278964Z 2022-11-23T03:04:48.5279048Z Running tests... 2022-11-23T03:04:48.5279313Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5279617Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100706 2022-11-23T03:04:48.5279825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100707 2022-11-23T03:04:48.5280203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5280409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5280794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5280969Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5281184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5281412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5281785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5281946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5282329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5282512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5282728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5282952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5283352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5283747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5284019Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5284293Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5284369Z ok (5.106s) 2022-11-23T03:04:48.5284384Z 2022-11-23T03:04:48.5284639Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5284740Z Ran 1 test in 5.106s 2022-11-23T03:04:48.5284745Z 2022-11-23T03:04:48.5284824Z OK 2022-11-23T03:04:48.5284829Z 2022-11-23T03:04:48.5284938Z Generating XML reports... 2022-11-23T03:04:48.5285360Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030337.xml 2022-11-23T03:04:48.5285672Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5286042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5286204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5286588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5286762Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5286772Z 2022-11-23T03:04:48.5286864Z Running tests... 2022-11-23T03:04:48.5287177Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5287479Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100918 2022-11-23T03:04:48.5287689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100919 2022-11-23T03:04:48.5288069Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5288229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5288611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5288786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5288998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5289273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5289652Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5289806Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5290189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5290367Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5290717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5290946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5291346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5291747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5292025Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5292299Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5292385Z ok (5.374s) 2022-11-23T03:04:48.5292391Z 2022-11-23T03:04:48.5292654Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5292748Z Ran 1 test in 5.374s 2022-11-23T03:04:48.5292754Z 2022-11-23T03:04:48.5292831Z OK 2022-11-23T03:04:48.5292836Z 2022-11-23T03:04:48.5292943Z Generating XML reports... 2022-11-23T03:04:48.5293367Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030346.xml 2022-11-23T03:04:48.5293678Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5294057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5294218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5294601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5294778Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5294784Z 2022-11-23T03:04:48.5294877Z Running tests... 2022-11-23T03:04:48.5295142Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5295425Z test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101134 2022-11-23T03:04:48.5295632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101135 2022-11-23T03:04:48.5296005Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5296237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5296631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5296807Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5297022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5297254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5297632Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5297793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5298175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5298410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5298626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5298849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5299251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5299646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5299731Z ok (4.809s) 2022-11-23T03:04:48.5299737Z 2022-11-23T03:04:48.5300000Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5300094Z Ran 1 test in 4.809s 2022-11-23T03:04:48.5300100Z 2022-11-23T03:04:48.5300175Z OK 2022-11-23T03:04:48.5300180Z 2022-11-23T03:04:48.5300288Z Generating XML reports... 2022-11-23T03:04:48.5300719Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030355.xml 2022-11-23T03:04:48.5301022Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5301393Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5301557Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5301940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5302116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5302121Z 2022-11-23T03:04:48.5302212Z Running tests... 2022-11-23T03:04:48.5302477Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5302766Z test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101339 2022-11-23T03:04:48.5302976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101340 2022-11-23T03:04:48.5303349Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5303511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5303895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5304074Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5304286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5304514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5304886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5305098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5305492Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5305669Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5305884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5306109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5306507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5306893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5307166Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5307492Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5307577Z ok (5.006s) 2022-11-23T03:04:48.5307583Z 2022-11-23T03:04:48.5307845Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5307943Z Ran 1 test in 5.007s 2022-11-23T03:04:48.5307949Z 2022-11-23T03:04:48.5308024Z OK 2022-11-23T03:04:48.5308030Z 2022-11-23T03:04:48.5308136Z Generating XML reports... 2022-11-23T03:04:48.5308561Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030404.xml 2022-11-23T03:04:48.5308873Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5309249Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5309407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5309793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5309973Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5309979Z 2022-11-23T03:04:48.5310072Z Running tests... 2022-11-23T03:04:48.5310333Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5310625Z test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101551 2022-11-23T03:04:48.5310833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101552 2022-11-23T03:04:48.5311203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5311363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5311748Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5311922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5312140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5312363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5312739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5312900Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5313281Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5313455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5313669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5313957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5314368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5314762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5315036Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5315309Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5315394Z ok (9.311s) 2022-11-23T03:04:48.5315400Z 2022-11-23T03:04:48.5315666Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5315761Z Ran 1 test in 9.312s 2022-11-23T03:04:48.5315767Z 2022-11-23T03:04:48.5315844Z OK 2022-11-23T03:04:48.5315850Z 2022-11-23T03:04:48.5315957Z Generating XML reports... 2022-11-23T03:04:48.5316385Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030413.xml 2022-11-23T03:04:48.5316752Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5317126Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5317288Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5317662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5317838Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5317844Z 2022-11-23T03:04:48.5317938Z Running tests... 2022-11-23T03:04:48.5318203Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5318484Z test_send_recv (__main__.ProcessGroupNCCLTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101763 2022-11-23T03:04:48.5318697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101764 2022-11-23T03:04:48.5319070Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5319231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5319615Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5319791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5320007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:04:48.5320233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:04:48.5320603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5320766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5321151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5321326Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5321543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:04:48.5321765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5322164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5322558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:04:48.5322832Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5323104Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:04:48.5323236Z ok (5.107s) 2022-11-23T03:04:48.5323242Z 2022-11-23T03:04:48.5323513Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5323610Z Ran 1 test in 5.107s 2022-11-23T03:04:48.5323616Z 2022-11-23T03:04:48.5323692Z OK 2022-11-23T03:04:48.5323698Z 2022-11-23T03:04:48.5323806Z Generating XML reports... 2022-11-23T03:04:48.5324230Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123030427.xml 2022-11-23T03:04:48.5324540Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5324918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5325080Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5325462Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5325691Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5325697Z 2022-11-23T03:04:48.5325789Z Running tests... 2022-11-23T03:04:48.5326056Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5326351Z test_common_errors (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5326751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:04:48.5326978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5327373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:04:48.5327598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5327997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:04:48.5328218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5328615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:04:48.5328700Z ok (0.611s) 2022-11-23T03:04:48.5328706Z 2022-11-23T03:04:48.5328969Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5329055Z Ran 1 test in 0.612s 2022-11-23T03:04:48.5329061Z 2022-11-23T03:04:48.5329139Z OK 2022-11-23T03:04:48.5329144Z 2022-11-23T03:04:48.5329251Z Generating XML reports... 2022-11-23T03:04:48.5329660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221123030436.xml 2022-11-23T03:04:48.5329979Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-11-23T03:04:48.5330353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:04:48.5330561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:04:48.5330947Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:04:48.5331122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:04:48.5331128Z 2022-11-23T03:04:48.5331220Z Running tests... 2022-11-23T03:04:48.5331482Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5331783Z test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5332239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:04:48.5332469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:04:48.5332870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:04:48.5332954Z ok (3.655s) 2022-11-23T03:04:48.5332960Z 2022-11-23T03:04:48.5333223Z ---------------------------------------------------------------------- 2022-11-23T03:04:48.5333319Z Ran 1 test in 3.655s 2022-11-23T03:04:48.5333325Z 2022-11-23T03:04:48.5333401Z OK 2022-11-23T03:04:48.5333407Z 2022-11-23T03:04:48.5333514Z Generating XML reports... 2022-11-23T03:04:48.5333904Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221123030440.xml 2022-11-23T03:04:48.5333910Z 2022-11-23T03:04:48.5334414Z ##[endgroup] 2022-11-23T03:04:48.5334895Z FINISHED PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_nccl_6kia0ius) 2022-11-23T03:04:48.5334971Z 2022-11-23T03:04:48.5335232Z Running distributed/test_c10d_error_logger ... [2022-11-23 03:04:48.391293] 2022-11-23T03:04:48.5335727Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_error_logger.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:04:48.391621] 2022-11-23T03:05:01.8182964Z 2022-11-23T03:05:01.8184520Z Expand the folded group to see the log file of distributed/test_c10d_error_logger 2022-11-23T03:05:01.8187619Z ##[group]PRINTING LOG FILE of distributed/test_c10d_error_logger (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_error_logger_9msxs_11) 2022-11-23T03:05:01.8190585Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_error_logger 2022-11-23T03:05:01.8191656Z 2022-11-23T03:05:01.8192031Z Running tests... 2022-11-23T03:05:01.8194406Z ---------------------------------------------------------------------- 2022-11-23T03:05:01.8197065Z test_exception_handler_with_dist (__main__.C10dErrorLoggerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102133 2022-11-23T03:05:01.8199214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102134 2022-11-23T03:05:01.8202668Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:05:01.8204374Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:05:01.8206649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:05:01.8208445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:05:01.8210130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:05:01.8212569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:05:01.8215770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:05:01.8218416Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:05:01.8221268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:05:01.8223111Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:05:01.8225192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:05:01.8227268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:05:01.8230153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:05:01.8233197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:05:01.8236368Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:01.8238560Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:01.8239874Z ok (5.016s) 2022-11-23T03:05:01.8241590Z test_get_or_create_logger (__main__.C10dErrorLoggerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102276 2022-11-23T03:05:01.8243753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102277 2022-11-23T03:05:01.8246663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:05:01.8248504Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:05:01.8251016Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:05:01.8253178Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:05:01.8255257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:05:01.8257832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:05:01.8260169Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:05:01.8262620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:05:01.8264570Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:05:01.8266369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:05:01.8267116Z ok (4.334s) 2022-11-23T03:05:01.8267366Z 2022-11-23T03:05:01.8267937Z ---------------------------------------------------------------------- 2022-11-23T03:05:01.8268466Z Ran 2 tests in 9.351s 2022-11-23T03:05:01.8268800Z 2022-11-23T03:05:01.8269023Z OK 2022-11-23T03:05:01.8269384Z 2022-11-23T03:05:01.8269710Z Generating XML reports... 2022-11-23T03:05:01.8271064Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_error_logger/TEST-C10dErrorLoggerTest-20221123030450.xml 2022-11-23T03:05:01.8272366Z 2022-11-23T03:05:01.8273378Z ##[endgroup] 2022-11-23T03:05:01.8274406Z FINISHED PRINTING LOG FILE of distributed/test_c10d_error_logger (/var/lib/jenkins/pytorch/test/test-reports/distributed-test_c10d_error_logger_9msxs_11) 2022-11-23T03:05:01.8274909Z 2022-11-23T03:05:01.8275349Z Running distributed/pipeline/sync/test_worker ... [2022-11-23 03:05:01.818157] 2022-11-23T03:05:01.8276436Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2022-11-23 03:05:01.818534] 2022-11-23T03:05:06.8874022Z 2022-11-23T03:05:06.8875137Z Expand the folded group to see the log file of distributed/pipeline/sync/test_worker 2022-11-23T03:05:06.8877954Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_worker_5q5bwy7w) 2022-11-23T03:05:06.8879651Z ============================= test session starts ============================== 2022-11-23T03:05:06.8881500Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:05:06.8882443Z cachedir: .pytest_cache 2022-11-23T03:05:06.8883970Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:05:06.8885061Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:05:06.8885885Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:05:06.8887388Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:05:06.8888385Z collecting ... collected 6 items 2022-11-23T03:05:06.8891523Z Running 6 items in this shard: test/distributed/pipeline/sync/test_worker.py::test_compute_multithreading, test/distributed/pipeline/sync/test_worker.py::test_compute_success, test/distributed/pipeline/sync/test_worker.py::test_compute_exception, test/distributed/pipeline/sync/test_worker.py::test_grad_mode[True], test/distributed/pipeline/sync/test_worker.py::test_grad_mode[False], test/distributed/pipeline/sync/test_worker.py::test_worker_per_device 2022-11-23T03:05:06.8893247Z 2022-11-23T03:05:06.8893733Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%] 2022-11-23T03:05:06.8894729Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED [ 33%] 2022-11-23T03:05:06.8895692Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED [ 50%] 2022-11-23T03:05:06.8896652Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED [ 66%] 2022-11-23T03:05:06.8897598Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED [ 83%] 2022-11-23T03:05:06.8898550Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED [100%] 2022-11-23T03:05:06.8899249Z 2022-11-23T03:05:06.8899573Z ============================== 6 passed in 0.04s =============================== 2022-11-23T03:05:06.8899972Z 2022-11-23T03:05:06.8900560Z ##[endgroup] 2022-11-23T03:05:06.8902016Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_worker_5q5bwy7w) 2022-11-23T03:05:06.8902817Z 2022-11-23T03:05:06.8903441Z Running distributed/pipeline/sync/test_stream ... [2022-11-23 03:05:06.887299] 2022-11-23T03:05:06.8904819Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-11-23 03:05:06.887667] 2022-11-23T03:05:12.0553015Z 2022-11-23T03:05:12.0554273Z Expand the folded group to see the log file of distributed/pipeline/sync/test_stream 2022-11-23T03:05:12.0557469Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_stream_syvezik9) 2022-11-23T03:05:12.0559155Z ============================= test session starts ============================== 2022-11-23T03:05:12.0560995Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:05:12.0562032Z cachedir: .pytest_cache 2022-11-23T03:05:12.0563878Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:05:12.0565154Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:05:12.0565986Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:05:12.0567509Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:05:12.0568514Z collecting ... collected 19 items 2022-11-23T03:05:12.0576557Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view 2022-11-23T03:05:12.0583270Z 2022-11-23T03:05:12.0583872Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [ 5%] 2022-11-23T03:05:12.0585133Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda PASSED [ 10%] 2022-11-23T03:05:12.0586422Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%] 2022-11-23T03:05:12.0587747Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda PASSED [ 21%] 2022-11-23T03:05:12.0589261Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%] 2022-11-23T03:05:12.0590562Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda PASSED [ 31%] 2022-11-23T03:05:12.0591841Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%] 2022-11-23T03:05:12.0593075Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda PASSED [ 42%] 2022-11-23T03:05:12.0594303Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%] 2022-11-23T03:05:12.0595531Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda PASSED [ 52%] 2022-11-23T03:05:12.0596751Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%] 2022-11-23T03:05:12.0597973Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda PASSED [ 63%] 2022-11-23T03:05:12.0599241Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%] 2022-11-23T03:05:12.0600536Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda PASSED [ 73%] 2022-11-23T03:05:12.0601824Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu PASSED [ 78%] 2022-11-23T03:05:12.0603116Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda PASSED [ 84%] 2022-11-23T03:05:12.0604417Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%] 2022-11-23T03:05:12.0605705Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda PASSED [ 94%] 2022-11-23T03:05:12.0607051Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view PASSED [100%] 2022-11-23T03:05:12.0607776Z 2022-11-23T03:05:12.0608150Z ============================== 19 passed in 1.23s ============================== 2022-11-23T03:05:12.0608635Z 2022-11-23T03:05:12.0609341Z ##[endgroup] 2022-11-23T03:05:12.0611228Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_stream_syvezik9) 2022-11-23T03:05:12.0612197Z 2022-11-23T03:05:12.0612959Z Running distributed/pipeline/sync/test_pipe ... [2022-11-23 03:05:12.055679] 2022-11-23T03:05:12.0614580Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-11-23 03:05:12.056325] 2022-11-23T03:05:22.6633422Z 2022-11-23T03:05:22.6635040Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipe 2022-11-23T03:05:22.6638062Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_pipe_zkj2b6j0) 2022-11-23T03:05:22.6640234Z ============================= test session starts ============================== 2022-11-23T03:05:22.6642379Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:05:22.6644170Z cachedir: .pytest_cache 2022-11-23T03:05:22.6646114Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:05:22.6647486Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:05:22.6648490Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:05:22.6650466Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:05:22.6651967Z collecting ... collected 56 items 2022-11-23T03:05:22.6668694Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper 2022-11-23T03:05:22.6684865Z 2022-11-23T03:05:22.6685426Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED [ 1%] 2022-11-23T03:05:22.6687283Z distributed/pipeline/sync/test_pipe.py::test_parameters libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6688361Z PASSED [ 3%] 2022-11-23T03:05:22.6690144Z distributed/pipeline/sync/test_pipe.py::test_public_attrs libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6691343Z PASSED [ 5%] 2022-11-23T03:05:22.6692900Z distributed/pipeline/sync/test_pipe.py::test_sequential_like libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6693963Z PASSED [ 7%] 2022-11-23T03:05:22.6695509Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6696585Z PASSED [ 8%] 2022-11-23T03:05:22.6698148Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6699313Z PASSED [ 10%] 2022-11-23T03:05:22.6700835Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6701904Z PASSED [ 12%] 2022-11-23T03:05:22.6703467Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6704551Z PASSED [ 14%] 2022-11-23T03:05:22.6706113Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6707210Z PASSED [ 16%] 2022-11-23T03:05:22.6708864Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6709980Z PASSED [ 17%] 2022-11-23T03:05:22.6711502Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6712586Z PASSED [ 19%] 2022-11-23T03:05:22.6714184Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6715287Z PASSED [ 21%] 2022-11-23T03:05:22.6716770Z distributed/pipeline/sync/test_pipe.py::test_no_grad libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6717833Z PASSED [ 23%] 2022-11-23T03:05:22.6719355Z distributed/pipeline/sync/test_pipe.py::test_exception libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6720435Z PASSED [ 25%] 2022-11-23T03:05:22.6722063Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6723178Z PASSED [ 26%] 2022-11-23T03:05:22.6724673Z distributed/pipeline/sync/test_pipe.py::test_nested_input libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6725749Z PASSED [ 28%] 2022-11-23T03:05:22.6727258Z distributed/pipeline/sync/test_pipe.py::test_input_pair libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6728509Z PASSED [ 30%] 2022-11-23T03:05:22.6730117Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6731530Z PASSED [ 32%] 2022-11-23T03:05:22.6733060Z distributed/pipeline/sync/test_pipe.py::test_input_singleton libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6734111Z PASSED [ 33%] 2022-11-23T03:05:22.6735629Z distributed/pipeline/sync/test_pipe.py::test_input_varargs libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6736711Z PASSED [ 35%] 2022-11-23T03:05:22.6738221Z distributed/pipeline/sync/test_pipe.py::test_non_tensor libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6739273Z PASSED [ 37%] 2022-11-23T03:05:22.6740856Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6742003Z PASSED [ 39%] 2022-11-23T03:05:22.6742703Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6743098Z PASSED [ 41%] 2022-11-23T03:05:22.6743663Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6744059Z PASSED [ 42%] 2022-11-23T03:05:22.6744640Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6745046Z PASSED [ 44%] 2022-11-23T03:05:22.6745602Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6746000Z PASSED [ 46%] 2022-11-23T03:05:22.6746571Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6747043Z PASSED [ 48%] 2022-11-23T03:05:22.6747620Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6748028Z PASSED [ 50%] 2022-11-23T03:05:22.6748595Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6748994Z PASSED [ 51%] 2022-11-23T03:05:22.6749569Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6749962Z PASSED [ 53%] 2022-11-23T03:05:22.6750548Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6750959Z PASSED [ 55%] 2022-11-23T03:05:22.6751508Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6751901Z PASSED [ 57%] 2022-11-23T03:05:22.6752455Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6752848Z PASSED [ 58%] 2022-11-23T03:05:22.6753406Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6753802Z PASSED [ 60%] 2022-11-23T03:05:22.6754379Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6754780Z PASSED [ 62%] 2022-11-23T03:05:22.6755357Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6755837Z PASSED [ 64%] 2022-11-23T03:05:22.6756441Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6756843Z PASSED [ 66%] 2022-11-23T03:05:22.6757432Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6757845Z PASSED [ 67%] 2022-11-23T03:05:22.6758436Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6758851Z PASSED [ 69%] 2022-11-23T03:05:22.6759385Z distributed/pipeline/sync/test_pipe.py::test_devices libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6759775Z PASSED [ 71%] 2022-11-23T03:05:22.6760328Z distributed/pipeline/sync/test_pipe.py::test_partitions libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6760778Z PASSED [ 73%] 2022-11-23T03:05:22.6761349Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6761746Z PASSED [ 75%] 2022-11-23T03:05:22.6762293Z distributed/pipeline/sync/test_pipe.py::test_deny_moving libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6762682Z PASSED [ 76%] 2022-11-23T03:05:22.6763225Z distributed/pipeline/sync/test_pipe.py::test_empty_module libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6763617Z PASSED [ 78%] 2022-11-23T03:05:22.6764177Z distributed/pipeline/sync/test_pipe.py::test_named_children libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6764569Z PASSED [ 80%] 2022-11-23T03:05:22.6765156Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6765565Z PASSED [ 82%] 2022-11-23T03:05:22.6766156Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6766561Z PASSED [ 83%] 2022-11-23T03:05:22.6767155Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6767567Z PASSED [ 85%] 2022-11-23T03:05:22.6768132Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6768531Z PASSED [ 87%] 2022-11-23T03:05:22.6769156Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6769596Z PASSED [ 89%] 2022-11-23T03:05:22.6770144Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6770595Z PASSED [ 91%] 2022-11-23T03:05:22.6771166Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6771566Z PASSED [ 92%] 2022-11-23T03:05:22.6772133Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6772538Z PASSED [ 94%] 2022-11-23T03:05:22.6773115Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6773514Z PASSED [ 96%] 2022-11-23T03:05:22.6774071Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6774534Z PASSED [ 98%] 2022-11-23T03:05:22.6775106Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:22.6775502Z PASSED [100%] 2022-11-23T03:05:22.6775644Z 2022-11-23T03:05:22.6775788Z ============================== 56 passed in 5.97s ============================== 2022-11-23T03:05:22.6775972Z 2022-11-23T03:05:22.6776252Z ##[endgroup] 2022-11-23T03:05:22.6776871Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_pipe_zkj2b6j0) 2022-11-23T03:05:22.6777227Z 2022-11-23T03:05:22.6777517Z Running distributed/pipeline/sync/test_microbatch ... [2022-11-23 03:05:22.664036] 2022-11-23T03:05:22.6778151Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2022-11-23 03:05:22.664637] 2022-11-23T03:05:26.6353154Z 2022-11-23T03:05:26.6354458Z Expand the folded group to see the log file of distributed/pipeline/sync/test_microbatch 2022-11-23T03:05:26.6356738Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_microbatch_f68yid65) 2022-11-23T03:05:26.6358146Z ============================= test session starts ============================== 2022-11-23T03:05:26.6359709Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:05:26.6360617Z cachedir: .pytest_cache 2022-11-23T03:05:26.6362169Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:05:26.6363302Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:05:26.6364142Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:05:26.6365677Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:05:26.6366720Z collecting ... collected 10 items 2022-11-23T03:05:26.6369697Z Running 10 items in this shard: test/distributed/pipeline/sync/test_microbatch.py::test_batch_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_call, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice, test/distributed/pipeline/sync/test_microbatch.py::test_check, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tensors, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tuples, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors 2022-11-23T03:05:26.6372670Z 2022-11-23T03:05:26.6373233Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED [ 10%] 2022-11-23T03:05:26.6374453Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%] 2022-11-23T03:05:26.6375647Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED [ 30%] 2022-11-23T03:05:26.6376860Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%] 2022-11-23T03:05:26.6378185Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%] 2022-11-23T03:05:26.6379479Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED [ 60%] 2022-11-23T03:05:26.6380704Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%] 2022-11-23T03:05:26.6381937Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED [ 80%] 2022-11-23T03:05:26.6383154Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%] 2022-11-23T03:05:26.6384441Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%] 2022-11-23T03:05:26.6385178Z 2022-11-23T03:05:26.6386140Z ============================== 10 passed in 0.06s ============================== 2022-11-23T03:05:26.6386656Z 2022-11-23T03:05:26.6387413Z ##[endgroup] 2022-11-23T03:05:26.6389333Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_microbatch_f68yid65) 2022-11-23T03:05:26.6390393Z 2022-11-23T03:05:26.6391225Z Running distributed/pipeline/sync/test_dependency ... [2022-11-23 03:05:26.635789] 2022-11-23T03:05:26.6393053Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-11-23 03:05:26.636405] 2022-11-23T03:05:30.9456234Z 2022-11-23T03:05:30.9457341Z Expand the folded group to see the log file of distributed/pipeline/sync/test_dependency 2022-11-23T03:05:30.9460047Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_dependency_waj3qc66) 2022-11-23T03:05:30.9462225Z ============================= test session starts ============================== 2022-11-23T03:05:30.9463791Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:05:30.9464837Z cachedir: .pytest_cache 2022-11-23T03:05:30.9466760Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:05:30.9468064Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:05:30.9468896Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:05:30.9470736Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:05:30.9471968Z collecting ... collected 6 items 2022-11-23T03:05:30.9474504Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad 2022-11-23T03:05:30.9476517Z 2022-11-23T03:05:30.9477057Z distributed/pipeline/sync/test_dependency.py::test_fork_join PASSED [ 16%] 2022-11-23T03:05:30.9478240Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%] 2022-11-23T03:05:30.9479412Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%] 2022-11-23T03:05:30.9480565Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED [ 66%] 2022-11-23T03:05:30.9481768Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%] 2022-11-23T03:05:30.9483038Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%] 2022-11-23T03:05:30.9483722Z 2022-11-23T03:05:30.9484096Z ============================== 6 passed in 0.37s =============================== 2022-11-23T03:05:30.9484573Z 2022-11-23T03:05:30.9485292Z ##[endgroup] 2022-11-23T03:05:30.9487058Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_dependency_waj3qc66) 2022-11-23T03:05:30.9488064Z 2022-11-23T03:05:30.9488819Z Running distributed/pipeline/sync/test_copy ... [2022-11-23 03:05:30.946086] 2022-11-23T03:05:30.9490430Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-11-23 03:05:30.946730] 2022-11-23T03:05:38.7769693Z 2022-11-23T03:05:38.7771901Z Expand the folded group to see the log file of distributed/pipeline/sync/test_copy 2022-11-23T03:05:38.7775320Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_copy_l9u2z678) 2022-11-23T03:05:38.7777929Z ============================= test session starts ============================== 2022-11-23T03:05:38.7780332Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:05:38.7781684Z cachedir: .pytest_cache 2022-11-23T03:05:38.7784184Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:05:38.7785897Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:05:38.7787165Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:05:38.7789732Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:05:38.7791287Z collecting ... collected 5 items 2022-11-23T03:05:38.7795033Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors 2022-11-23T03:05:38.7798494Z 2022-11-23T03:05:38.7799428Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED [ 20%] 2022-11-23T03:05:38.7801386Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda PASSED [ 40%] 2022-11-23T03:05:38.7803417Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu PASSED [ 60%] 2022-11-23T03:05:38.7805290Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda PASSED [ 80%] 2022-11-23T03:05:38.7807216Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%] 2022-11-23T03:05:38.7808282Z 2022-11-23T03:05:38.7808857Z ============================== 5 passed in 3.84s =============================== 2022-11-23T03:05:38.7809605Z 2022-11-23T03:05:38.7810927Z ##[endgroup] 2022-11-23T03:05:38.7814125Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_copy_l9u2z678) 2022-11-23T03:05:38.7815928Z 2022-11-23T03:05:38.7817135Z Running distributed/pipeline/sync/test_bugs ... [2022-11-23 03:05:38.777717] 2022-11-23T03:05:38.7819890Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2022-11-23 03:05:38.778630] 2022-11-23T03:05:45.1138550Z 2022-11-23T03:05:45.1140032Z Expand the folded group to see the log file of distributed/pipeline/sync/test_bugs 2022-11-23T03:05:45.1142301Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_bugs_m6vf43gu) 2022-11-23T03:05:45.1144034Z ============================= test session starts ============================== 2022-11-23T03:05:45.1145617Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:05:45.1146533Z cachedir: .pytest_cache 2022-11-23T03:05:45.1148069Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:05:45.1149177Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:05:45.1150006Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:05:45.1151507Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:05:45.1152522Z collecting ... collected 4 items 2022-11-23T03:05:45.1154283Z Running 4 items in this shard: test/distributed/pipeline/sync/test_bugs.py::test_python_autograd_function, test/distributed/pipeline/sync/test_bugs.py::test_exception_no_hang, test/distributed/pipeline/sync/test_bugs.py::test_tuple_wait, test/distributed/pipeline/sync/test_bugs.py::test_parallel_randoms 2022-11-23T03:05:45.1155662Z 2022-11-23T03:05:45.1156857Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:45.1158382Z PASSED [ 25%] 2022-11-23T03:05:45.1159909Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:45.1160968Z PASSED [ 50%] 2022-11-23T03:05:45.1162429Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:45.1163445Z PASSED [ 75%] 2022-11-23T03:05:45.1164971Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:45.1166021Z PASSED [100%] 2022-11-23T03:05:45.1166394Z 2022-11-23T03:05:45.1166767Z ============================== 4 passed in 2.16s =============================== 2022-11-23T03:05:45.1167242Z 2022-11-23T03:05:45.1167952Z ##[endgroup] 2022-11-23T03:05:45.1169614Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-test_bugs_m6vf43gu) 2022-11-23T03:05:45.1170962Z 2022-11-23T03:05:45.1171843Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-11-23 03:05:45.114183] 2022-11-23T03:05:45.1173618Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-11-23 03:05:45.114810] 2022-11-23T03:05:49.1500485Z 2022-11-23T03:05:49.1502141Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_verify_skippables 2022-11-23T03:05:49.1505619Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_we6ys7hx) 2022-11-23T03:05:49.1507613Z ============================= test session starts ============================== 2022-11-23T03:05:49.1510042Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:05:49.1511274Z cachedir: .pytest_cache 2022-11-23T03:05:49.1513171Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:05:49.1514501Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:05:49.1515527Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:05:49.1517733Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:05:49.1518847Z collecting ... collected 9 items 2022-11-23T03:05:49.1522682Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated 2022-11-23T03:05:49.1526010Z 2022-11-23T03:05:49.1526694Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%] 2022-11-23T03:05:49.1528048Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%] 2022-11-23T03:05:49.1529378Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%] 2022-11-23T03:05:49.1531088Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%] 2022-11-23T03:05:49.1532338Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%] 2022-11-23T03:05:49.1534177Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%] 2022-11-23T03:05:49.1536218Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%] 2022-11-23T03:05:49.1538221Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%] 2022-11-23T03:05:49.1540006Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%] 2022-11-23T03:05:49.1540921Z 2022-11-23T03:05:49.1541424Z ============================== 9 passed in 0.04s =============================== 2022-11-23T03:05:49.1542056Z 2022-11-23T03:05:49.1542904Z ##[endgroup] 2022-11-23T03:05:49.1545012Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_we6ys7hx) 2022-11-23T03:05:49.1546151Z 2022-11-23T03:05:49.1546978Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-11-23 03:05:49.150567] 2022-11-23T03:05:49.1549120Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-11-23 03:05:49.151189] 2022-11-23T03:05:53.2452684Z 2022-11-23T03:05:53.2454036Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_stash_pop 2022-11-23T03:05:53.2456888Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_t1fzkunl) 2022-11-23T03:05:53.2458352Z ============================= test session starts ============================== 2022-11-23T03:05:53.2459981Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:05:53.2460978Z cachedir: .pytest_cache 2022-11-23T03:05:53.2462537Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:05:53.2463759Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:05:53.2464645Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:05:53.2466225Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:05:53.2467293Z collecting ... collected 7 items 2022-11-23T03:05:53.2469917Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none 2022-11-23T03:05:53.2472101Z 2022-11-23T03:05:53.2472655Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED [ 14%] 2022-11-23T03:05:53.2473856Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED [ 28%] 2022-11-23T03:05:53.2475074Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%] 2022-11-23T03:05:53.2476337Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%] 2022-11-23T03:05:53.2477589Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%] 2022-11-23T03:05:53.2478801Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%] 2022-11-23T03:05:53.2480043Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%] 2022-11-23T03:05:53.2480704Z 2022-11-23T03:05:53.2481104Z ============================== 7 passed in 0.06s =============================== 2022-11-23T03:05:53.2481607Z 2022-11-23T03:05:53.2482368Z ##[endgroup] 2022-11-23T03:05:53.2484672Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_t1fzkunl) 2022-11-23T03:05:53.2485757Z 2022-11-23T03:05:53.2486587Z Running distributed/pipeline/sync/skip/test_leak ... [2022-11-23 03:05:53.245717] 2022-11-23T03:05:53.2488302Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2022-11-23 03:05:53.246349] 2022-11-23T03:05:57.6608493Z 2022-11-23T03:05:57.6610257Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_leak 2022-11-23T03:05:57.6612792Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_leak_ar0imrne) 2022-11-23T03:05:57.6614273Z ============================= test session starts ============================== 2022-11-23T03:05:57.6615901Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:05:57.6617392Z cachedir: .pytest_cache 2022-11-23T03:05:57.6619050Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:05:57.6620247Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:05:57.6621148Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:05:57.6622717Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:05:57.6623827Z collecting ... collected 8 items 2022-11-23T03:05:57.6628655Z Running 8 items in this shard: test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] 2022-11-23T03:05:57.6631230Z 2022-11-23T03:05:57.6632361Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:57.6633349Z PASSED [ 12%] 2022-11-23T03:05:57.6634803Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:57.6635833Z PASSED [ 25%] 2022-11-23T03:05:57.6637315Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:57.6638348Z PASSED [ 37%] 2022-11-23T03:05:57.6639828Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:57.6640863Z PASSED [ 50%] 2022-11-23T03:05:57.6642267Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:57.6643594Z PASSED [ 62%] 2022-11-23T03:05:57.6645243Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:57.6646441Z PASSED [ 75%] 2022-11-23T03:05:57.6648101Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:57.6649269Z PASSED [ 87%] 2022-11-23T03:05:57.6676436Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:05:57.6677744Z PASSED [100%] 2022-11-23T03:05:57.6678086Z 2022-11-23T03:05:57.6678411Z ============================== 8 passed in 0.32s =============================== 2022-11-23T03:05:57.6678820Z 2022-11-23T03:05:57.6679489Z ##[endgroup] 2022-11-23T03:05:57.6681060Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_leak_ar0imrne) 2022-11-23T03:05:57.6681925Z 2022-11-23T03:05:57.6682592Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-11-23 03:05:57.661156] 2022-11-23T03:05:57.6684046Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-11-23 03:05:57.661782] 2022-11-23T03:06:06.9963399Z 2022-11-23T03:06:06.9964474Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_gpipe 2022-11-23T03:06:06.9967412Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_3imkxg91) 2022-11-23T03:06:06.9969363Z ============================= test session starts ============================== 2022-11-23T03:06:06.9971102Z platform linux -- Python 3.8.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-11-23T03:06:06.9972014Z cachedir: .pytest_cache 2022-11-23T03:06:06.9974166Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/pytorch/test/.hypothesis/examples') 2022-11-23T03:06:06.9975284Z torch: 1.14.0a0+git1cfd385 2022-11-23T03:06:06.9976122Z rootdir: /var/lib/jenkins/pytorch, configfile: pytest.ini 2022-11-23T03:06:06.9977659Z plugins: flakefinder-1.1.0, hypothesis-5.35.1, shard-0.1.2, rerunfailures-10.3, xdoctest-1.0.2, xdist-3.0.2 2022-11-23T03:06:06.9978682Z collecting ... collected 13 items 2022-11-23T03:06:06.9985026Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip 2022-11-23T03:06:06.9988605Z 2022-11-23T03:06:06.9989774Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:06.9990858Z PASSED [ 7%] 2022-11-23T03:06:06.9992369Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:06.9993438Z PASSED [ 15%] 2022-11-23T03:06:06.9994935Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:06.9995991Z PASSED [ 23%] 2022-11-23T03:06:06.9997487Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:06.9998552Z SKIPPED [ 30%] 2022-11-23T03:06:07.0000064Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:07.0001117Z PASSED [ 38%] 2022-11-23T03:06:07.0002896Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:07.0003985Z PASSED [ 46%] 2022-11-23T03:06:07.0005508Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:07.0006545Z PASSED [ 53%] 2022-11-23T03:06:07.0008067Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:07.0009140Z SKIPPED [ 61%] 2022-11-23T03:06:07.0010979Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:07.0012318Z PASSED [ 69%] 2022-11-23T03:06:07.0013902Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:07.0015191Z PASSED [ 76%] 2022-11-23T03:06:07.0016739Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:07.0017817Z PASSED [ 84%] 2022-11-23T03:06:07.0019353Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:07.0020424Z SKIPPED [ 92%] 2022-11-23T03:06:07.0021905Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:06:07.0022954Z PASSED [100%] 2022-11-23T03:06:07.0023342Z 2022-11-23T03:06:07.0023752Z ======================== 10 passed, 3 skipped in 4.32s ========================= 2022-11-23T03:06:07.0024264Z 2022-11-23T03:06:07.0024974Z ##[endgroup] 2022-11-23T03:06:07.0026737Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/pytorch/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_3imkxg91) 2022-11-23T03:06:07.0027767Z 2022-11-23T03:06:07.0028581Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-11-23 03:06:06.996495] 2022-11-23T03:06:07.0030606Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:06:06.997206] 2022-11-23T03:09:34.5918068Z 2022-11-23T03:09:34.5919386Z Expand the folded group to see the log file of distributed/optim/test_zero_redundancy_optimizer 2022-11-23T03:09:34.5922457Z ##[group]PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/pytorch/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_1a27b4y_) 2022-11-23T03:09:34.5925594Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer 2022-11-23T03:09:34.5926593Z 2022-11-23T03:09:34.5926948Z Running tests... 2022-11-23T03:09:34.5928487Z ---------------------------------------------------------------------- 2022-11-23T03:09:34.5929972Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.5934430Z Check that ZeroRedundancyOptimizer properly handles adding a new ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67287 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.569s) 2022-11-23T03:09:34.5937401Z test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.5939253Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106889 2022-11-23T03:09:34.5941284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106890 2022-11-23T03:09:34.5944719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.5946249Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.5948187Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.5949924Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.5951675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.5954284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.5956100Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.5958421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.5960238Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.5963013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.5964650Z skip: Test skipped for ROCm (4.933s) 2022-11-23T03:09:34.5966616Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.5969149Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107023 2022-11-23T03:09:34.5971632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107024 2022-11-23T03:09:34.5974158Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.5975892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.5978300Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.5980107Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.5981765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.5983916Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.5985312Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.5987193Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.5988652Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.5990020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.5991199Z skip: Test skipped for ROCm (4.930s) 2022-11-23T03:09:34.5992878Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.5995075Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107157 2022-11-23T03:09:34.5996789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107158 2022-11-23T03:09:34.5998780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6000174Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6002039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6003493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6004833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6007170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6008576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6010465Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6012075Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6013333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6014420Z skip: Test skipped for ROCm (4.331s) 2022-11-23T03:09:34.6015945Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6017991Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107291 2022-11-23T03:09:34.6019809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107292 2022-11-23T03:09:34.6021681Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6022973Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6024702Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6025983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6026870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6028163Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6029074Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6030314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6031271Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6032154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6032927Z skip: Test skipped for ROCm (4.331s) 2022-11-23T03:09:34.6034018Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6035438Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107425 2022-11-23T03:09:34.6036537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107426 2022-11-23T03:09:34.6037832Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6038746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6039959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6040909Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6041795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6043102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6043991Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6045225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6046171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6047194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6047969Z skip: Test skipped for ROCm (4.130s) 2022-11-23T03:09:34.6049058Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6050476Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107559 2022-11-23T03:09:34.6051904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107560 2022-11-23T03:09:34.6053220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6054125Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6055345Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6056443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6057331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6058645Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6059542Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6060737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6061684Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6062563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6063332Z skip: Test skipped for ROCm (4.832s) 2022-11-23T03:09:34.6064418Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6065605Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107693 2022-11-23T03:09:34.6066252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107694 2022-11-23T03:09:34.6066995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6067509Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6068221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6068766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6069283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6070038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6070565Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6071272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6071826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6072331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6072774Z skip: Test skipped for ROCm (4.231s) 2022-11-23T03:09:34.6073402Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6074222Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107827 2022-11-23T03:09:34.6074948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107828 2022-11-23T03:09:34.6075704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6076229Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6076921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6077471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6077987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6078731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6079254Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6080032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6080592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6081102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6081530Z skip: Test skipped for ROCm (4.631s) 2022-11-23T03:09:34.6082162Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6082987Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107961 2022-11-23T03:09:34.6083630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107962 2022-11-23T03:09:34.6084374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6084905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6085609Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6086148Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6086662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6087411Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6087932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6088634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6089182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6089698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6090147Z skip: Test skipped for ROCm (4.328s) 2022-11-23T03:09:34.6090864Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6091697Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108095 2022-11-23T03:09:34.6092339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108096 2022-11-23T03:09:34.6093086Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6093608Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6094310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6095055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6095556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6096322Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6096847Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6097548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6098101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6098612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6099065Z skip: Test skipped for ROCm (4.831s) 2022-11-23T03:09:34.6099706Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6100579Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108229 2022-11-23T03:09:34.6101231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108230 2022-11-23T03:09:34.6101977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6102506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6103210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6103758Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6104267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6105031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6105548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6106260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6106721Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6107150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6107526Z skip: Test skipped for ROCm (4.732s) 2022-11-23T03:09:34.6108049Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6108728Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108363 2022-11-23T03:09:34.6109260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108364 2022-11-23T03:09:34.6109876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6110316Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6110898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6111360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6111785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6112412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6112853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6113477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6113945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6114378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6114752Z skip: Test skipped for ROCm (4.732s) 2022-11-23T03:09:34.6115275Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6115993Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108497 2022-11-23T03:09:34.6116639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108498 2022-11-23T03:09:34.6117379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6117971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6118677Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6119229Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6119745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6120505Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6121034Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6121733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6122270Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6122787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6123249Z skip: Test skipped for ROCm (4.753s) 2022-11-23T03:09:34.6123879Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6124703Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108631 2022-11-23T03:09:34.6125350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108632 2022-11-23T03:09:34.6126096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6126614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6127320Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6127877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6128385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6129147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6129673Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6130374Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6131203Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6131741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6132187Z skip: Test skipped for ROCm (4.731s) 2022-11-23T03:09:34.6132969Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6133890Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108765 2022-11-23T03:09:34.6134544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108766 2022-11-23T03:09:34.6135316Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6135841Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6136534Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6137087Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6137597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6138354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6138950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6139660Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6140209Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6140729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6141173Z skip: Test skipped for ROCm (4.332s) 2022-11-23T03:09:34.6141792Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6142604Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108899 2022-11-23T03:09:34.6143265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108900 2022-11-23T03:09:34.6144012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6144539Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6145240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6145803Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6146309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6146957Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6147396Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6147981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6148446Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6148872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6149244Z skip: Test skipped for ROCm (4.840s) 2022-11-23T03:09:34.6149753Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6150434Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109033 2022-11-23T03:09:34.6150976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109034 2022-11-23T03:09:34.6151599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6152043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6152707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6153171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6153598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6154221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6154656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6155242Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6155702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6156141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6156573Z skip: Test skipped for ROCm (5.030s) 2022-11-23T03:09:34.6157025Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6157620Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109167 2022-11-23T03:09:34.6158152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109168 2022-11-23T03:09:34.6158776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6159216Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6159803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6160266Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6160702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6161166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:09:34.6161800Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6162237Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6162820Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6163273Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6163700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6164177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6164845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6165535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6166064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhd8s85m 2022-11-23T03:09:34.6166587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhd8s85m/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6167103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjs1i4_dw 2022-11-23T03:09:34.6167623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjs1i4_dw/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6168191Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6168699Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6169772Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6171428Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6172667Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6173557Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6174460Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6175345Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6176231Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6177110Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6177991Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6178860Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6179723Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6180597Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6181484Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6182356Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6183235Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6184108Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6184984Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6186603Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6187088Z ok (7.442s) 2022-11-23T03:09:34.6187502Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6188111Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109320 2022-11-23T03:09:34.6188645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109321 2022-11-23T03:09:34.6189275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6189721Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6190312Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6190833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6191267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6191750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6192389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6192814Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6193399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6193862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6194292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6194778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:09:34.6195444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6196145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6196678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qhqmfx4 2022-11-23T03:09:34.6197187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qhqmfx4/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6197702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2851lvxk 2022-11-23T03:09:34.6198219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2851lvxk/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6198777Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6199290Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6200287Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6201863Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6202982Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6203717Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6204446Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6205174Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6205948Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6206670Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6207394Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6208119Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6208829Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6209561Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6210283Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6211075Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6211796Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6212518Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6213391Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6214261Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6214813Z ok (7.541s) 2022-11-23T03:09:34.6215306Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6216039Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109473 2022-11-23T03:09:34.6216666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109474 2022-11-23T03:09:34.6217431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6218058Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6218775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6219330Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6219847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6220417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:09:34.6221185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6221709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6222412Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6222968Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6223558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6224132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6224932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6225773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6226399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8gyjv2mn 2022-11-23T03:09:34.6226961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8gyjv2mn/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6227474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_7rvp13 2022-11-23T03:09:34.6227987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_7rvp13/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6228552Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6229057Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6230042Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6231557Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6232682Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6233412Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6234137Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6234910Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6235636Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6236362Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6237079Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6237789Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6238511Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6239283Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6240008Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6240732Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6241456Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6242183Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6242910Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6243631Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6244093Z ok (7.640s) 2022-11-23T03:09:34.6244502Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6245101Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109626 2022-11-23T03:09:34.6245634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109627 2022-11-23T03:09:34.6246280Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6246728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6247324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6247786Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6248221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6248687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6249326Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6249770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6250415Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6251014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6251444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6251925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:09:34.6252601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6253405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6254038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4doc7kf 2022-11-23T03:09:34.6254666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4doc7kf/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6255369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdvi99qy 2022-11-23T03:09:34.6255994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdvi99qy/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6256668Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6257278Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6258478Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6260305Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6261665Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6262538Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6263419Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6264296Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6265163Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6266015Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6266820Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6267604Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6268342Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6269067Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6269792Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6270513Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6271236Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6272016Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6272736Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6273455Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6273913Z ok (7.539s) 2022-11-23T03:09:34.6274311Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6274913Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109779 2022-11-23T03:09:34.6275451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109780 2022-11-23T03:09:34.6276091Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6276535Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6277127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6277588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6278020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6278486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:09:34.6279122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6279575Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6280166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6280638Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6281071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6281549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6282204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6282905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6283444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9w4sit9 2022-11-23T03:09:34.6284018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9w4sit9/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6284536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdk77g5o 2022-11-23T03:09:34.6285056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdk77g5o/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6285621Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6286129Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6287107Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6288700Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6289808Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6290584Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6291314Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6292040Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6292750Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6293470Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6294191Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6294917Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6295640Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6296359Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6297079Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6297800Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6298587Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6299312Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6300030Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6300749Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6301205Z ok (7.647s) 2022-11-23T03:09:34.6301599Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6302254Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109932 2022-11-23T03:09:34.6302786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109933 2022-11-23T03:09:34.6303429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6303872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6304456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6304917Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6305333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6305807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:09:34.6306451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6306894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6307485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6307952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6308383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6308852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6309519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6310215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6310755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjqlvy9a 2022-11-23T03:09:34.6311272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjqlvy9a/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6311785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5y9incm 2022-11-23T03:09:34.6312305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5y9incm/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6312863Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6313353Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6314399Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6315923Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:09:34.6317032Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6317799Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6318527Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6319243Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6319966Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6320692Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6321423Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6322148Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6322868Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6323590Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6324315Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6325044Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6325764Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6326484Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6327213Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6327932Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-11-23T03:09:34.6328380Z ok (7.339s) 2022-11-23T03:09:34.6328769Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6329308Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110085 2022-11-23T03:09:34.6329830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110086 2022-11-23T03:09:34.6330467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6330966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6331555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6332004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6332438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6332993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6333633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6334083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6334670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6335135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6335555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6336035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:09:34.6336695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6337399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6337971Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6338475Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6338793Z ok (5.632s) 2022-11-23T03:09:34.6339132Z test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6339706Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110236 2022-11-23T03:09:34.6340255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110237 2022-11-23T03:09:34.6340879Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6341323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6341922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6342388Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6342820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6343284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:09:34.6343928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6344370Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6344958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6345418Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6345922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6346404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6347059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6347754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6348328Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6348835Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6349150Z ok (6.937s) 2022-11-23T03:09:34.6349506Z test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6350256Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110389 2022-11-23T03:09:34.6350874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110390 2022-11-23T03:09:34.6351484Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6351925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6352510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6352974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6353402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6353920Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4 2022-11-23T03:09:34.6354590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6355024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6355610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6356070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6356497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6357010Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4 2022-11-23T03:09:34.6357391Z ok (4.842s) 2022-11-23T03:09:34.6357720Z test_sharding (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6358890Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67295 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.003s) 2022-11-23T03:09:34.6359691Z test_step (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6360230Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110523 2022-11-23T03:09:34.6360776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110524 2022-11-23T03:09:34.6361397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6361838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6362426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6362890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6363388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6364011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6364455Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6365042Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6365502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6365930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6366306Z skip: Test skipped for ROCm (4.930s) 2022-11-23T03:09:34.6366684Z test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6367230Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110657 2022-11-23T03:09:34.6367816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110658 2022-11-23T03:09:34.6368441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6368880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6369464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6369926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6370353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6371100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6371547Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6372144Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6372601Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6373028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6373403Z skip: Test skipped for ROCm (4.231s) 2022-11-23T03:09:34.6373776Z test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6374300Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110791 2022-11-23T03:09:34.6374830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110792 2022-11-23T03:09:34.6375453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6375897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6376494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6376958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6377390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6378018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6378444Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6379036Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6379499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6379929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6380493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:09:34.6380979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6381645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6382322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6382851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpti5loajj 2022-11-23T03:09:34.6383374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpti5loajj/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6383896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpro9fnhb0 2022-11-23T03:09:34.6384416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpro9fnhb0/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6384975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:09:34.6385448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:09:34.6386110Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T03:09:34.6386543Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T03:09:34.6387133Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T03:09:34.6387577Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T03:09:34.6387854Z ok (5.564s) 2022-11-23T03:09:34.6388186Z test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6388719Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110997 2022-11-23T03:09:34.6389258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110998 2022-11-23T03:09:34.6389868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6390311Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6390895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6391354Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6391789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6392261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:09:34.6392903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6393338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6393936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6394399Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6394832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6395307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6395970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6396664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:09:34.6397196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp042vg5va 2022-11-23T03:09:34.6397802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp042vg5va/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6398322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkxl8q1u 2022-11-23T03:09:34.6398844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkxl8q1u/_remote_module_non_scriptable.py 2022-11-23T03:09:34.6399412Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6399915Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6400363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:09:34.6400834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:09:34.6401479Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T03:09:34.6401926Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T03:09:34.6402577Z /opt/conda/lib/python3.8/tempfile.py:818: ResourceWarning: Implicitly cleaning up 2022-11-23T03:09:34.6403024Z _warnings.warn(warn_message, ResourceWarning) 2022-11-23T03:09:34.6403305Z ok (7.133s) 2022-11-23T03:09:34.6403704Z test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6404464Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111150 2022-11-23T03:09:34.6404984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111151 2022-11-23T03:09:34.6405608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6406050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6406650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6407114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6407549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:09:34.6408180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6408610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6409195Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6409659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6410094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6410481Z skip: Need at least 4 CUDA devices (4.530s) 2022-11-23T03:09:34.6411023Z test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-11-23T03:09:34.6412232Z Check that ZeRO works with model parallelism where the model's ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75984 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.003s) 2022-11-23T03:09:34.6413013Z test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T03:09:34.6413566Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111284 2022-11-23T03:09:34.6414268Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6414713Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6415376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6415841Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6416272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6416751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6417418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:09:34.6417783Z ok (4.123s) 2022-11-23T03:09:34.6418116Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T03:09:34.6418643Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111352 2022-11-23T03:09:34.6419340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6419877Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6420466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6420927Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6421362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6421827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6422490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:09:34.6423061Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6423376Z ok (4.423s) 2022-11-23T03:09:34.6423725Z test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T03:09:34.6424301Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111428 2022-11-23T03:09:34.6425018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6425448Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6426039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6426499Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6426928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6427402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6428065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:09:34.6428448Z ok (3.921s) 2022-11-23T03:09:34.6428761Z test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T03:09:34.6429320Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111496 2022-11-23T03:09:34.6430045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6430489Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6431076Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6431538Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6431971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6432515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6433184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:09:34.6433756Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6434077Z ok (5.326s) 2022-11-23T03:09:34.6434421Z test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T03:09:34.6434995Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111572 2022-11-23T03:09:34.6435718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6436161Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6436733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6437260Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6437693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6438172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6438838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:09:34.6439410Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6439723Z ok (4.424s) 2022-11-23T03:09:34.6440047Z test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T03:09:34.6440591Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111648 2022-11-23T03:09:34.6441304Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6441750Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6442339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6442799Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6443234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6443700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6444363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:09:34.6444934Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6445249Z ok (4.423s) 2022-11-23T03:09:34.6445601Z test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T03:09:34.6446145Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111724 2022-11-23T03:09:34.6446846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6447284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6447858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6448324Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6448756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6449232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6449973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:09:34.6450670Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:09:34.6450989Z ok (4.522s) 2022-11-23T03:09:34.6451304Z test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-11-23T03:09:34.6451822Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111800 2022-11-23T03:09:34.6452518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:09:34.6452962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:09:34.6453551Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:09:34.6454013Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:09:34.6454525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:09:34.6454991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:09:34.6455660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:09:34.6456041Z ok (4.123s) 2022-11-23T03:09:34.6456182Z 2022-11-23T03:09:34.6456458Z ---------------------------------------------------------------------- 2022-11-23T03:09:34.6456786Z Ran 42 tests in 203.475s 2022-11-23T03:09:34.6456943Z 2022-11-23T03:09:34.6457045Z OK (skipped=23) 2022-11-23T03:09:34.6457193Z 2022-11-23T03:09:34.6457305Z Generating XML reports... 2022-11-23T03:09:34.6458014Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20221123030608.xml 2022-11-23T03:09:34.6458985Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20221123030608.xml 2022-11-23T03:09:34.6459409Z 2022-11-23T03:09:34.6459873Z ##[endgroup] 2022-11-23T03:09:34.6460556Z FINISHED PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/pytorch/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_1a27b4y_) 2022-11-23T03:09:34.6460943Z 2022-11-23T03:09:34.6461224Z Running distributed/nn/jit/test_instantiator ... [2022-11-23 03:09:34.594044] 2022-11-23T03:09:34.6461932Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:09:34.594727] 2022-11-23T03:09:39.4951949Z 2022-11-23T03:09:39.4952865Z Expand the folded group to see the log file of distributed/nn/jit/test_instantiator 2022-11-23T03:09:39.4955626Z ##[group]PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/pytorch/test/test-reports/distributed-nn-jit-test_instantiator_qgkwfkld) 2022-11-23T03:09:39.4958443Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator 2022-11-23T03:09:39.4959385Z 2022-11-23T03:09:39.4959717Z Running tests... 2022-11-23T03:09:39.4960896Z ---------------------------------------------------------------------- 2022-11-23T03:09:39.4962018Z test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (0.603s) 2022-11-23T03:09:39.4963218Z test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s) 2022-11-23T03:09:39.4964659Z test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.014s) 2022-11-23T03:09:39.4965330Z 2022-11-23T03:09:39.4966069Z ---------------------------------------------------------------------- 2022-11-23T03:09:39.4966910Z Ran 3 tests in 0.619s 2022-11-23T03:09:39.4967308Z 2022-11-23T03:09:39.4967513Z OK 2022-11-23T03:09:39.4967828Z 2022-11-23T03:09:39.4968138Z Generating XML reports... 2022-11-23T03:09:39.4970235Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20221123030936.xml 2022-11-23T03:09:39.4971500Z 2022-11-23T03:09:39.4972241Z ##[endgroup] 2022-11-23T03:09:39.4973932Z FINISHED PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/pytorch/test/test-reports/distributed-nn-jit-test_instantiator_qgkwfkld) 2022-11-23T03:09:39.4974871Z 2022-11-23T03:09:39.4975577Z Running distributed/fsdp/test_utils ... [2022-11-23 03:09:39.495543] 2022-11-23T03:09:39.4977408Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:09:39.496229] 2022-11-23T03:09:44.0352449Z 2022-11-23T03:09:44.0353113Z Expand the folded group to see the log file of distributed/fsdp/test_utils 2022-11-23T03:09:44.0355123Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_utils__wwerwd_) 2022-11-23T03:09:44.0357561Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils 2022-11-23T03:09:44.0358275Z 2022-11-23T03:09:44.0358531Z Running tests... 2022-11-23T03:09:44.0359643Z ---------------------------------------------------------------------- 2022-11-23T03:09:44.0360702Z test_module_wrap_policy (__main__.TestGetSubmoduleToStates) 2022-11-23T03:09:44.0361768Z Tests the module wrap policy on a nested model with buffers and a ... ok (0.597s) 2022-11-23T03:09:44.0362817Z test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... ok (0.005s) 2022-11-23T03:09:44.0364129Z test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s) 2022-11-23T03:09:44.0365420Z test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... ok (0.004s) 2022-11-23T03:09:44.0366347Z test_packed_sequence (__main__.TestUtils) 2022-11-23T03:09:44.0367314Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.004s) 2022-11-23T03:09:44.0368350Z test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s) 2022-11-23T03:09:44.0368879Z 2022-11-23T03:09:44.0369616Z ---------------------------------------------------------------------- 2022-11-23T03:09:44.0370462Z Ran 6 tests in 0.615s 2022-11-23T03:09:44.0371050Z 2022-11-23T03:09:44.0371262Z OK 2022-11-23T03:09:44.0371570Z 2022-11-23T03:09:44.0371863Z Generating XML reports... 2022-11-23T03:09:44.0373520Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20221123030941.xml 2022-11-23T03:09:44.0375513Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221123030941.xml 2022-11-23T03:09:44.0376340Z 2022-11-23T03:09:44.0377055Z ##[endgroup] 2022-11-23T03:09:44.0378612Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_utils__wwerwd_) 2022-11-23T03:09:44.0379479Z 2022-11-23T03:09:44.0380297Z Running distributed/fsdp/test_fsdp_use_orig_params ... [2022-11-23 03:09:44.035879] 2022-11-23T03:09:44.0382250Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:09:44.036575] 2022-11-23T03:13:34.3436593Z 2022-11-23T03:13:34.3438272Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_use_orig_params 2022-11-23T03:13:34.3440703Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_u78b6tc5) 2022-11-23T03:13:34.3443263Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params 2022-11-23T03:13:34.3444059Z 2022-11-23T03:13:34.3444796Z Running tests... 2022-11-23T03:13:34.3446489Z ---------------------------------------------------------------------- 2022-11-23T03:13:34.3448054Z test_named_parameters_in_forward (__main__.TestFSDPUseOrigParamsFQNs) 2022-11-23T03:13:34.3451664Z Tests that calling ``named_parameters()`` during forward returns FQNs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112069 2022-11-23T03:13:34.3453595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112070 2022-11-23T03:13:34.3455972Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3457240Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3459407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3460895Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3462436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3464649Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3466431Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3468228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3469719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3471323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3473690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3476121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3478059Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3479838Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3481367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3482861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3483955Z dist init r=0, world=2 2022-11-23T03:13:34.3484742Z dist init r=1, world=2 2022-11-23T03:13:34.3485485Z ok (7.036s) 2022-11-23T03:13:34.3486479Z test_param_and_buffer_names (__main__.TestFSDPUseOrigParamsFQNs) 2022-11-23T03:13:34.3488202Z Tests that, for ``use_orig_params=True``, the parameter and buffer ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112214 2022-11-23T03:13:34.3490053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112215 2022-11-23T03:13:34.3492450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3493996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3496020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3497510Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3498883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3500890Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3502150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3503826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3505117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3506329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3509447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3511442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3513057Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3514487Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3515747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3517057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3518042Z dist init r=0, world=2 2022-11-23T03:13:34.3521508Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:13:34.3523769Z warnings.warn( 2022-11-23T03:13:34.3524366Z dist init r=1, world=2 2022-11-23T03:13:34.3527600Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:13:34.3529631Z warnings.warn( 2022-11-23T03:13:34.3530213Z ok (4.430s) 2022-11-23T03:13:34.3531991Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T03:13:34.3533652Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112357 2022-11-23T03:13:34.3535082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112358 2022-11-23T03:13:34.3536757Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3537964Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3539514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3540740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3541909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3543601Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3544766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3546324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3547163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3547670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3548323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3549008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3549570Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3550067Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3550592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3551051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3551509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3551973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3552318Z dist init r=0, world=2 2022-11-23T03:13:34.3552556Z dist init r=1, world=2 2022-11-23T03:13:34.3552781Z ok (7.835s) 2022-11-23T03:13:34.3553189Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T03:13:34.3553798Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112510 2022-11-23T03:13:34.3554378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112511 2022-11-23T03:13:34.3555000Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3555434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3556012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3556464Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3556894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3557520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3557948Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3558527Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3558985Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3559416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3560065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3560754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3561317Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3561811Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3562235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3562691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3563161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3563621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3563965Z dist init r=0, world=2 2022-11-23T03:13:34.3564201Z dist init r=1, world=2 2022-11-23T03:13:34.3564417Z ok (7.638s) 2022-11-23T03:13:34.3564837Z test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T03:13:34.3565457Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112663 2022-11-23T03:13:34.3565991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112664 2022-11-23T03:13:34.3566607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3567046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3567684Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3568145Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3568570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3569200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3569629Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3570207Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3570729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3571174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3572037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3572840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3573522Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3574118Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3574642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3575193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3575754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3576317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3576722Z dist init r=0, world=2 2022-11-23T03:13:34.3577003Z dist init r=1, world=2 2022-11-23T03:13:34.3577271Z ok (7.545s) 2022-11-23T03:13:34.3577744Z test_diff_hyperparams_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T03:13:34.3578468Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112816 2022-11-23T03:13:34.3579106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112817 2022-11-23T03:13:34.3579845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3580353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3581051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3581598Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3582125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3582883Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3583399Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3584095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3584627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3585144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3585922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3586712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3587336Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3587837Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3588273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3588731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3589187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3589651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3590117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3590577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3591038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3591553Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3592018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3592470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3592927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3593385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3593851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3594308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3594767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3595230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3595682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3596142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3596606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3597067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3597525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3597985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3598443Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3598897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3599361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3599816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3600274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3600730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3601185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3601642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3602090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3602548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3603006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3603521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3603979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3604438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3604898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3605359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3605809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3606267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3606726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3607182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3607686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3608144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3608603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3609054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3609513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3609969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3610424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3610966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3612506Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:13:34.3613419Z warnings.warn( 2022-11-23T03:13:34.3613838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3615416Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:13:34.3616329Z warnings.warn( 2022-11-23T03:13:34.3616743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3617298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3617855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3618411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3618963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3619515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3620065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3620603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3621261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3621819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3622371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3622920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3623474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3624023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3624564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3625113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3625669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3626284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3626806Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3627265Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3627723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3628171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3628627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3629087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3629544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3630013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3630478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3630934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3631390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3631838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3632294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3632751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3633214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3633674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3634137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3634594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3635042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3635497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3635953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3636411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3636877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3637336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3637795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3638302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3638766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3639229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3639686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3640029Z dist init r=1, world=2 2022-11-23T03:13:34.3640264Z dist init r=0, world=2 2022-11-23T03:13:34.3640479Z ok (39.079s) 2022-11-23T03:13:34.3640875Z test_diff_hyperparams_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T03:13:34.3641473Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112969 2022-11-23T03:13:34.3642002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112970 2022-11-23T03:13:34.3642682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3643119Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3643700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3644155Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3644580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3645212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3645642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3646220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3646678Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3647109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3647756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3648436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3649001Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3649498Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3649935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3650397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3651330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3652025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3652562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3653101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3653647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3654196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3654748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3655303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3655853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3656487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3657043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3657594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3658139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3658693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3659247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3659798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3660352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3660897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3662229Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3663782Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3665336Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3666813Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3668088Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3669370Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3670653Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3671925Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3673196Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3674529Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3675245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3675711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3676168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3676629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3677087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3677547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3678054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3678506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3678966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3679426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3679884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3680340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3680800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3681258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3681722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3682170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3682628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3683093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3683550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3684009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3684467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3684927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3685376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3685841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3686871Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3688136Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3689457Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3690808Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3692242Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3693747Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3695351Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3696877Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3698413Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3699939Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3700796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3701351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3701902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3702454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3703007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3703570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3705140Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:13:34.3706066Z warnings.warn( 2022-11-23T03:13:34.3706489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3707900Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:13:34.3708667Z warnings.warn( 2022-11-23T03:13:34.3709023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3709487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3709949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3710406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3710866Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3711326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3711824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3712286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3712746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3713205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3713673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3714134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3714592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3715042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3715501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3715965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3716427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3717456Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3718720Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3719996Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3721264Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3722522Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3723842Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3725110Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3726372Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3727638Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3728946Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3729663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3730127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3730671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3731143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3731612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3732073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3732531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3732994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3733454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3733909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3734372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3734831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3735293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3735763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3736226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3736685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3737137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3737600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3738060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3738518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3738982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3739444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3739968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3740421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3741457Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3742722Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3743983Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3745322Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3746584Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3747850Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3749107Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3750382Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3751646Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3752910Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3753627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3754085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3754550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3755013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3755533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3755997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3756344Z dist init r=1, world=2 2022-11-23T03:13:34.3756574Z dist init r=0, world=2 2022-11-23T03:13:34.3756804Z ok (40.293s) 2022-11-23T03:13:34.3757203Z test_diff_hyperparams_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T03:13:34.3757810Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113122 2022-11-23T03:13:34.3758345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113123 2022-11-23T03:13:34.3758971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3759461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3760051Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3760502Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3760940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3761573Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3762010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3762596Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3763049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3763489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3764146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3764838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3765407Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3765905Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3766343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3766804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3767266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3767720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3768189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3768651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3769115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3769577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3770041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3770500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3771074Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3771535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3771995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3772525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3772987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3773447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3773906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3774364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3774814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3775275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3775731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3776240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3776699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3777159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3777618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3778066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3778527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3779049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3779592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3780134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3780688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3781229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3781765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3782308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3782853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3783392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3783935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3784475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3785027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3785574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3786131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3786688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3787239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3787789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3788339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3788887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3789426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3789976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3790578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3791130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3792736Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:13:34.3793661Z warnings.warn( 2022-11-23T03:13:34.3794089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3795669Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:13:34.3796651Z warnings.warn( 2022-11-23T03:13:34.3797027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3797481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3797943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3798401Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3798863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3799329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3799793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3800250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3800708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3801157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3801614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3802073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3802529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3802987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3803452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3803912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3804361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3804818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3805275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3805735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3806193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3806653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3807156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3807612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3808070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3808530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3808997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3809456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3809913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3810374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3810882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3811402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3811862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3812322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3812781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3813240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3813701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3814151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3814612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3815070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3815534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3815991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3816450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3816914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3817364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3817822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3818283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3818624Z dist init r=0, world=2 2022-11-23T03:13:34.3818861Z dist init r=1, world=2 2022-11-23T03:13:34.3819093Z ok (39.088s) 2022-11-23T03:13:34.3819433Z test_diff_trainability (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T03:13:34.3820000Z Tests FSDP parity with DDP when using multiple parameter groups and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113275 2022-11-23T03:13:34.3820538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113276 2022-11-23T03:13:34.3821171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3821614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3822198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3822657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3823094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3823770Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3824219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3824805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3825261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3825699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3826354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3827043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3827609Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3828099Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3828679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3829139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3829481Z dist init r=0, world=2 2022-11-23T03:13:34.3829718Z dist init r=1, world=2 2022-11-23T03:13:34.3829947Z ok (10.641s) 2022-11-23T03:13:34.3830286Z test_multiple_optimizers (__main__.TestFSDPUseOrigParamsMultipleParamGroups) 2022-11-23T03:13:34.3830842Z Tests using two optimizers where only one sets gradients to ``None``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113428 2022-11-23T03:13:34.3831364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113429 2022-11-23T03:13:34.3831987Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3832423Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3833014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3833473Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3833899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3834529Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3834962Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3835539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3835994Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3836431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3837091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3837780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3838339Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3838836Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3839274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3839736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3840804Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:13:34.3842346Z [W reducer.cpp:1298] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-11-23T03:13:34.3843203Z dist init r=1, world=2 2022-11-23T03:13:34.3843438Z dist init r=0, world=2 2022-11-23T03:13:34.3843668Z ok (7.433s) 2022-11-23T03:13:34.3844045Z test_access_params_after_forward (__main__.TestFSDPUseOrigParamsParamAccess) 2022-11-23T03:13:34.3844596Z Tests that accessing the original parameters after the forward but ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113581 2022-11-23T03:13:34.3845131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113582 2022-11-23T03:13:34.3845744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3846182Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3846766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3847219Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3847655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3848292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3848728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3849309Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3849755Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3850188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3850884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3851584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3852150Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3852655Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3853093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3853545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3854006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3854467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3854929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3855391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3855854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3856315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3856654Z dist init r=0, world=2 2022-11-23T03:13:34.3856962Z dist init r=1, world=2 2022-11-23T03:13:34.3857191Z ok (6.730s) 2022-11-23T03:13:34.3857551Z test_multiple_forward_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T03:13:34.3858113Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113734 2022-11-23T03:13:34.3858635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113735 2022-11-23T03:13:34.3859263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3859692Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3860270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3860722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3861214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3861846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3862281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3862865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3863312Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3863743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3864394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3865087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3865660Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3866158Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3866593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3867053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3867384Z dist init r=1, world=2 2022-11-23T03:13:34.3867620Z dist init r=0, world=2 2022-11-23T03:13:34.3867849Z ok (8.036s) 2022-11-23T03:13:34.3868207Z test_multiple_forward_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T03:13:34.3868760Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113887 2022-11-23T03:13:34.3869276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113888 2022-11-23T03:13:34.3869892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3870332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3870918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3871381Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3871818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3872452Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3872887Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3873456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3874166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3874605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3875268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3875957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3876522Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3877018Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3877457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3877904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3878241Z dist init r=1, world=2 2022-11-23T03:13:34.3878525Z dist init r=0, world=2 2022-11-23T03:13:34.3878759Z ok (8.042s) 2022-11-23T03:13:34.3879134Z test_summon_between_two_forwards_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T03:13:34.3879704Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114040 2022-11-23T03:13:34.3880216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114041 2022-11-23T03:13:34.3880839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3881275Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3881857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3882313Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3882757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3883392Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3883826Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3884396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3884851Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3885286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3885941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3886630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3887200Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3887699Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3888127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3888587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3888929Z dist init r=0, world=2 2022-11-23T03:13:34.3889168Z dist init r=1, world=2 2022-11-23T03:13:34.3889393Z ok (8.137s) 2022-11-23T03:13:34.3889768Z test_summon_between_two_forwards_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard) 2022-11-23T03:13:34.3890333Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114193 2022-11-23T03:13:34.3890954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114194 2022-11-23T03:13:34.3891651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3892099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3892683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3893137Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3893571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3894201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3894627Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3895209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3895731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3896166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3896823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3897514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3898078Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3898573Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3899002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3899463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3900501Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3901789Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3903062Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3904341Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:13:34.3904942Z dist init r=0, world=2 2022-11-23T03:13:34.3905183Z dist init r=1, world=2 2022-11-23T03:13:34.3905410Z ok (8.236s) 2022-11-23T03:13:34.3905725Z test_grad_writeback (__main__.TestFSDPUseOrigParamsWriteback) 2022-11-23T03:13:34.3906407Z Tests that changes to the original parameters' gradients are written ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114346 2022-11-23T03:13:34.3906944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114347 2022-11-23T03:13:34.3907560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3907998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3908636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3909094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3909532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3910170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3910599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3911182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3911635Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3912067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3912782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3913469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3914035Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3914524Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3914963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3915424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3915889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3916356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3916823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3917295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3917757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3918210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3918674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3919135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3919598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3920058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3920517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3920985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:13:34.3921318Z dist init r=0, world=2 2022-11-23T03:13:34.3921555Z dist init r=1, world=2 2022-11-23T03:13:34.3921778Z ok (7.034s) 2022-11-23T03:13:34.3922091Z test_param_writeback (__main__.TestFSDPUseOrigParamsWriteback) 2022-11-23T03:13:34.3922613Z Tests that changes to the original parameters are written back. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114499 2022-11-23T03:13:34.3923136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114500 2022-11-23T03:13:34.3923744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3924185Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3924766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3925276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3925711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3926342Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3926782Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3927352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3927806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3928239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3928889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3929579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3930202Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3930757Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3931201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3931653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3931990Z dist init r=1, world=2 2022-11-23T03:13:34.3932227Z dist init r=0, world=2 2022-11-23T03:13:34.3932451Z ok (4.528s) 2022-11-23T03:13:34.3932908Z test_writeback_shape_mismatch (__main__.TestFSDPUseOrigParamsWriteback) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114646 2022-11-23T03:13:34.3933460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114647 2022-11-23T03:13:34.3934083Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3934519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3935099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3935558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3935989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:34.3936618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:34.3937047Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:34.3937628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:34.3938078Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:34.3938517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:34.3939167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3939851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:34.3940417Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3940914Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:34.3941352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:34.3941804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:34.3942147Z dist init r=0, world=2 2022-11-23T03:13:34.3942381Z dist init r=1, world=2 2022-11-23T03:13:34.3942612Z ok (4.530s) 2022-11-23T03:13:34.3942806Z 2022-11-23T03:13:34.3943087Z ---------------------------------------------------------------------- 2022-11-23T03:13:34.3943406Z Ran 18 tests in 226.299s 2022-11-23T03:13:34.3943558Z 2022-11-23T03:13:34.3943627Z OK 2022-11-23T03:13:34.3943746Z 2022-11-23T03:13:34.3943856Z Generating XML reports... 2022-11-23T03:13:34.3944508Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20221123030945.xml 2022-11-23T03:13:34.3945411Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20221123030945.xml 2022-11-23T03:13:34.3946325Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20221123030945.xml 2022-11-23T03:13:34.3947227Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20221123030945.xml 2022-11-23T03:13:34.3948180Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20221123030945.xml 2022-11-23T03:13:34.3948557Z 2022-11-23T03:13:34.3948948Z ##[endgroup] 2022-11-23T03:13:34.3949582Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_u78b6tc5) 2022-11-23T03:13:34.3949945Z 2022-11-23T03:13:34.3950221Z Running distributed/fsdp/test_fsdp_traversal ... [2022-11-23 03:13:34.346613] 2022-11-23T03:13:34.3951161Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:13:34.347284] 2022-11-23T03:13:43.5081018Z 2022-11-23T03:13:43.5082934Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_traversal 2022-11-23T03:13:43.5085688Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_traversal__hajrxtw) 2022-11-23T03:13:43.5088111Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal 2022-11-23T03:13:43.5088863Z 2022-11-23T03:13:43.5089104Z Running tests... 2022-11-23T03:13:43.5090205Z ---------------------------------------------------------------------- 2022-11-23T03:13:43.5092095Z test_fsdp_modules (__main__.TestTraversal) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114856 2022-11-23T03:13:43.5093553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114857 2022-11-23T03:13:43.5095248Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:43.5096417Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:43.5097968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:43.5099201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:43.5100347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:13:43.5102023Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:13:43.5103171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:13:43.5104705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:13:43.5105902Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:13:43.5107050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:13:43.5109269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:43.5110915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:13:43.5112184Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:43.5113296Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:13:43.5114269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:13:43.5115298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:13:43.5116057Z dist init r=0, world=2 2022-11-23T03:13:43.5116565Z dist init r=1, world=2 2022-11-23T03:13:43.5117060Z ok (5.169s) 2022-11-23T03:13:43.5117357Z 2022-11-23T03:13:43.5117961Z ---------------------------------------------------------------------- 2022-11-23T03:13:43.5118649Z Ran 1 test in 5.170s 2022-11-23T03:13:43.5119139Z 2022-11-23T03:13:43.5119306Z OK 2022-11-23T03:13:43.5119573Z 2022-11-23T03:13:43.5119813Z Generating XML reports... 2022-11-23T03:13:43.5121127Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221123031336.xml 2022-11-23T03:13:43.5121848Z 2022-11-23T03:13:43.5122448Z ##[endgroup] 2022-11-23T03:13:43.5123822Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_traversal__hajrxtw) 2022-11-23T03:13:43.5124603Z 2022-11-23T03:13:43.5125283Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-11-23 03:13:43.508298] 2022-11-23T03:13:43.5126927Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:13:43.508962] 2022-11-23T03:18:13.7211916Z 2022-11-23T03:18:13.7212888Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_summon_full_params 2022-11-23T03:18:13.7217586Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_iej_4uxc) 2022-11-23T03:18:13.7223673Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params 2022-11-23T03:18:13.7224541Z 2022-11-23T03:18:13.7224804Z Running tests... 2022-11-23T03:18:13.7225968Z ---------------------------------------------------------------------- 2022-11-23T03:18:13.7227453Z test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115066 2022-11-23T03:18:13.7229003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115067 2022-11-23T03:18:13.7231299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7232544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7234233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7235506Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7236709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7238477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7239675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7241673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7243284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7244784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7247840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7250421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7253303Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7255164Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7256692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7259291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7260449Z dist init r=0, world=2 2022-11-23T03:18:13.7264710Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7267859Z warnings.warn( 2022-11-23T03:18:13.7268696Z dist init r=1, world=2 2022-11-23T03:18:13.7273117Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7275807Z warnings.warn( 2022-11-23T03:18:13.7276630Z ok (7.076s) 2022-11-23T03:18:13.7278164Z test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115215 2022-11-23T03:18:13.7279992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115216 2022-11-23T03:18:13.7282159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7283667Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7285784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7287332Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7288941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7291444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7292755Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7294375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7295609Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7296769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7298552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7300316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7301440Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7302392Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7303254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7304329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7304991Z dist init r=1, world=2 2022-11-23T03:18:13.7305441Z dist init r=0, world=2 2022-11-23T03:18:13.7307878Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7309380Z warnings.warn( 2022-11-23T03:18:13.7311794Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7313368Z warnings.warn( 2022-11-23T03:18:13.7313785Z ok (4.426s) 2022-11-23T03:18:13.7314422Z test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams) 2022-11-23T03:18:13.7315436Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115358 2022-11-23T03:18:13.7316427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115359 2022-11-23T03:18:13.7317626Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7318468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7319614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7320476Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7321311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7322524Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7323353Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7324473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7325341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7326166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7327438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7328754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7329850Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7330891Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7331741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7332635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7333295Z dist init r=1, world=2 2022-11-23T03:18:13.7333747Z dist init r=0, world=2 2022-11-23T03:18:13.7334166Z ok (4.428s) 2022-11-23T03:18:13.7334812Z test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams) 2022-11-23T03:18:13.7335950Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115501 2022-11-23T03:18:13.7336962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115502 2022-11-23T03:18:13.7338171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7339007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7340134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7340995Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7341823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7343032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7343968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7345100Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7345974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7346803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7348068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7349402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7350498Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7351458Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7352310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7353213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7353874Z dist init r=1, world=2 2022-11-23T03:18:13.7354310Z dist init r=0, world=2 2022-11-23T03:18:13.7354740Z ok (4.527s) 2022-11-23T03:18:13.7355423Z test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams) 2022-11-23T03:18:13.7356476Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115648 2022-11-23T03:18:13.7357467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115649 2022-11-23T03:18:13.7358675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7359506Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7360634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7361514Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7362351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7363577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7364414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7365530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7366414Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7367229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7368508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7369968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7371147Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7372107Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7372943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7373845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7374498Z dist init r=0, world=2 2022-11-23T03:18:13.7374950Z dist init r=1, world=2 2022-11-23T03:18:13.7375381Z ok (4.626s) 2022-11-23T03:18:13.7376060Z test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams) 2022-11-23T03:18:13.7377112Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115791 2022-11-23T03:18:13.7378239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115792 2022-11-23T03:18:13.7379449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7380272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7381401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7382277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7383102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7384324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7385149Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7386279Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7387140Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7387959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7389224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7390575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7391674Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7392636Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7393476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7394375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7395010Z dist init r=1, world=2 2022-11-23T03:18:13.7395457Z dist init r=0, world=2 2022-11-23T03:18:13.7395887Z ok (4.527s) 2022-11-23T03:18:13.7396847Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115938 2022-11-23T03:18:13.7398002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115939 2022-11-23T03:18:13.7399212Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7400042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7401151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7402139Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7402978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7404214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7405038Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7406172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7407049Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7407864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7409127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7410456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7411913Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7412876Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7413742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7414647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7415299Z dist init r=0, world=2 2022-11-23T03:18:13.7415747Z dist init r=1, world=2 2022-11-23T03:18:13.7416171Z ok (4.428s) 2022-11-23T03:18:13.7417136Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116081 2022-11-23T03:18:13.7418291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116082 2022-11-23T03:18:13.7419509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7420153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7420843Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7421392Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7421920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7422683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7423202Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7423899Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7424449Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7424953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7425750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7426576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7427256Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7427855Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7428380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7428933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7429342Z dist init r=1, world=2 2022-11-23T03:18:13.7429618Z dist init r=0, world=2 2022-11-23T03:18:13.7429966Z ok (5.130s) 2022-11-23T03:18:13.7430524Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116224 2022-11-23T03:18:13.7431122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116225 2022-11-23T03:18:13.7431741Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7432172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7432751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7433196Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7433629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7434318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7434748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7435325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7435776Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7436204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7436846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7437532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7438094Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7438597Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7439031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7439488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7439826Z dist init r=1, world=2 2022-11-23T03:18:13.7440850Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7441495Z warnings.warn( 2022-11-23T03:18:13.7442496Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7443135Z warnings.warn( 2022-11-23T03:18:13.7443366Z dist init r=0, world=2 2022-11-23T03:18:13.7444368Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7445024Z warnings.warn( 2022-11-23T03:18:13.7446061Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7446705Z warnings.warn( 2022-11-23T03:18:13.7446925Z ok (5.327s) 2022-11-23T03:18:13.7447420Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116367 2022-11-23T03:18:13.7448008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116368 2022-11-23T03:18:13.7448628Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7449061Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7449640Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7450099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7450640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7451278Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7451764Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7452450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7452998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7453508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7454302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7455132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7455818Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7456419Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7456933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7457487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7457894Z dist init r=1, world=2 2022-11-23T03:18:13.7459120Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7459904Z warnings.warn( 2022-11-23T03:18:13.7461103Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7461881Z warnings.warn( 2022-11-23T03:18:13.7462157Z dist init r=0, world=2 2022-11-23T03:18:13.7463374Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7464150Z warnings.warn( 2022-11-23T03:18:13.7465404Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7466173Z warnings.warn( 2022-11-23T03:18:13.7466436Z ok (4.527s) 2022-11-23T03:18:13.7467039Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116510 2022-11-23T03:18:13.7467748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116511 2022-11-23T03:18:13.7468498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7469020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7469727Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7470244Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7470676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7471313Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7471746Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7472323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7472779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7473216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7473873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7474557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7475125Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7475624Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7476059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7476518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7476862Z dist init r=1, world=2 2022-11-23T03:18:13.7477088Z dist init r=0, world=2 2022-11-23T03:18:13.7477317Z ok (4.428s) 2022-11-23T03:18:13.7477812Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116655 2022-11-23T03:18:13.7478408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116656 2022-11-23T03:18:13.7479026Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7479461Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7480035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7480492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7480914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7481542Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7481970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7482545Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7483067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7483497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7484147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7484823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7485387Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7485882Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7486317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7486772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7487233Z dist init r=1, world=2 2022-11-23T03:18:13.7487470Z dist init r=0, world=2 2022-11-23T03:18:13.7487685Z ok (4.526s) 2022-11-23T03:18:13.7488188Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116800 2022-11-23T03:18:13.7488779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116801 2022-11-23T03:18:13.7489402Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7489834Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7490414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7490913Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7491371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7492140Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7492659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7493350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7493898Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7494421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7495211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7496037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7496714Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7497310Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7497835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7498391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7498804Z dist init r=1, world=2 2022-11-23T03:18:13.7499088Z dist init r=0, world=2 2022-11-23T03:18:13.7499344Z ok (4.629s) 2022-11-23T03:18:13.7499945Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116945 2022-11-23T03:18:13.7500656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116946 2022-11-23T03:18:13.7501397Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7502000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7502718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7503269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7503795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7504541Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7505066Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7505773Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7506322Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7506923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7507715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7508547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7509218Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7509823Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7510272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7510736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7511084Z dist init r=1, world=2 2022-11-23T03:18:13.7511329Z dist init r=0, world=2 2022-11-23T03:18:13.7511559Z ok (4.528s) 2022-11-23T03:18:13.7512058Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117090 2022-11-23T03:18:13.7512655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117091 2022-11-23T03:18:13.7513277Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7513710Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7514287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7514742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7515176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7515793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7516231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7516807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7517258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7517689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7518338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7519020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7519581Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7520065Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7520566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7521025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7521365Z dist init r=0, world=2 2022-11-23T03:18:13.7521600Z dist init r=1, world=2 2022-11-23T03:18:13.7521827Z ok (5.530s) 2022-11-23T03:18:13.7522312Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117237 2022-11-23T03:18:13.7522909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117238 2022-11-23T03:18:13.7523526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7523957Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7524538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7525047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7525481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7526108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7526530Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7527108Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7527564Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7527998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7528648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7529333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7529897Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7530381Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7530944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7531443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7531845Z dist init r=0, world=2 2022-11-23T03:18:13.7532127Z dist init r=1, world=2 2022-11-23T03:18:13.7532394Z ok (4.531s) 2022-11-23T03:18:13.7532991Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117384 2022-11-23T03:18:13.7533707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117385 2022-11-23T03:18:13.7534459Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7534982Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7535678Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7536227Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7536746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7537502Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7538021Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7538876Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7539424Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7539946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7540730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7541557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7542236Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7542831Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7543341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7543889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7544373Z dist init r=0, world=2 2022-11-23T03:18:13.7544655Z dist init r=1, world=2 2022-11-23T03:18:13.7544923Z ok (4.528s) 2022-11-23T03:18:13.7545512Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117527 2022-11-23T03:18:13.7546220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117528 2022-11-23T03:18:13.7546954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7547472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7548162Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7548705Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7549231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7549989Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7550424Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7550993Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7551445Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7551876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7552522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7553206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7553777Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7554280Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7554706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7555171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7555514Z dist init r=0, world=2 2022-11-23T03:18:13.7555761Z dist init r=1, world=2 2022-11-23T03:18:13.7555991Z ok (4.427s) 2022-11-23T03:18:13.7556494Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117670 2022-11-23T03:18:13.7557093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117671 2022-11-23T03:18:13.7557782Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7558231Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7558821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7559284Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7559722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7560352Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7560788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7561359Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7561817Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7562318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7562978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7563666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7564236Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7564740Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7565185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7565633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7565979Z dist init r=0, world=2 2022-11-23T03:18:13.7566220Z dist init r=1, world=2 2022-11-23T03:18:13.7566455Z ok (4.727s) 2022-11-23T03:18:13.7566958Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117817 2022-11-23T03:18:13.7567554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117818 2022-11-23T03:18:13.7568172Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7568601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7569186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7569650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7570087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7570783Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7571217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7571918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7572455Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7572982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7573782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7574622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7575297Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7575981Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7576520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7577062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7577475Z dist init r=0, world=2 2022-11-23T03:18:13.7577762Z dist init r=1, world=2 2022-11-23T03:18:13.7578037Z ok (4.627s) 2022-11-23T03:18:13.7578636Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117964 2022-11-23T03:18:13.7579339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117965 2022-11-23T03:18:13.7580088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7580599Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7581372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7581920Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7582445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7583199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7583717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7584418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7584964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7585476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7586272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7587099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7587775Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7588377Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7588909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7589462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7589865Z dist init r=1, world=2 2022-11-23T03:18:13.7590128Z dist init r=0, world=2 2022-11-23T03:18:13.7590355Z ok (4.829s) 2022-11-23T03:18:13.7590856Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118109 2022-11-23T03:18:13.7592014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118110 2022-11-23T03:18:13.7592631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7593068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7593636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7594097Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7594531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7595164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7595595Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7596240Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7596702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7597128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7597790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7598476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7599040Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7599539Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7599974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7600490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7600829Z dist init r=1, world=2 2022-11-23T03:18:13.7601059Z dist init r=0, world=2 2022-11-23T03:18:13.7601286Z ok (4.522s) 2022-11-23T03:18:13.7601587Z test_raises_rank0_with_writeback (__main__.TestSummonFullParams) 2022-11-23T03:18:13.7602086Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118254 2022-11-23T03:18:13.7602596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118255 2022-11-23T03:18:13.7603214Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7603639Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7604218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7604679Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7605111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7605736Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7606168Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7606746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7607201Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7607625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7608276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7608969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7609531Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7610027Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7610463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7610974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7611304Z dist init r=0, world=2 2022-11-23T03:18:13.7611541Z dist init r=1, world=2 2022-11-23T03:18:13.7611766Z ok (4.525s) 2022-11-23T03:18:13.7612295Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118397 2022-11-23T03:18:13.7612987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118398 2022-11-23T03:18:13.7613610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7614045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7614616Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7615070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7615503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7616128Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7616561Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7617143Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7617651Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7618075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7618733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7619414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7619973Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7620469Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7620907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7621364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7621711Z dist init r=0, world=2 2022-11-23T03:18:13.7621935Z dist init r=1, world=2 2022-11-23T03:18:13.7623150Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7623922Z warnings.warn( 2022-11-23T03:18:13.7624901Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7625481Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7626721Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7627488Z warnings.warn( 2022-11-23T03:18:13.7628285Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7628857Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7629115Z ok (6.430s) 2022-11-23T03:18:13.7629697Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118546 2022-11-23T03:18:13.7630324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118547 2022-11-23T03:18:13.7630944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7631383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7631966Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7632422Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7632857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7633483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7633970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7634555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7635010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7635439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7636093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7636778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7637340Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7637837Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7638267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7638728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7639071Z dist init r=0, world=2 2022-11-23T03:18:13.7640281Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7641038Z warnings.warn( 2022-11-23T03:18:13.7641844Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7642419Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7642688Z dist init r=1, world=2 2022-11-23T03:18:13.7643882Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7644634Z warnings.warn( 2022-11-23T03:18:13.7645421Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7645983Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7646244Z ok (7.135s) 2022-11-23T03:18:13.7646827Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118699 2022-11-23T03:18:13.7647458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118700 2022-11-23T03:18:13.7648081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7648517Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7649099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7649543Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7649976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7650809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7651246Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7651831Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7652280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7652714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7653354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7654046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7654612Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7655120Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7655570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7656028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7656367Z dist init r=1, world=2 2022-11-23T03:18:13.7656594Z dist init r=0, world=2 2022-11-23T03:18:13.7657797Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7658562Z warnings.warn( 2022-11-23T03:18:13.7659361Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7659931Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7660981Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7661638Z warnings.warn( 2022-11-23T03:18:13.7662697Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7663350Z warnings.warn( 2022-11-23T03:18:13.7664558Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7665312Z warnings.warn( 2022-11-23T03:18:13.7666106Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7666671Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7667771Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7668422Z warnings.warn( 2022-11-23T03:18:13.7669398Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7670038Z warnings.warn( 2022-11-23T03:18:13.7670263Z ok (6.633s) 2022-11-23T03:18:13.7670791Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118848 2022-11-23T03:18:13.7671413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118849 2022-11-23T03:18:13.7672030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7672465Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7673047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7673493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7673933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7674564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7675000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7675579Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7676033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7676469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7677126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7677804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7678371Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7678867Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7679385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7679849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7680189Z dist init r=0, world=2 2022-11-23T03:18:13.7681399Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7682164Z warnings.warn( 2022-11-23T03:18:13.7682958Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7683568Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7684616Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7685263Z warnings.warn( 2022-11-23T03:18:13.7686253Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7686900Z warnings.warn( 2022-11-23T03:18:13.7687137Z dist init r=1, world=2 2022-11-23T03:18:13.7688343Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7689100Z warnings.warn( 2022-11-23T03:18:13.7689895Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7690457Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7691554Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7692207Z warnings.warn( 2022-11-23T03:18:13.7693181Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7693823Z warnings.warn( 2022-11-23T03:18:13.7694043Z ok (6.631s) 2022-11-23T03:18:13.7694568Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119001 2022-11-23T03:18:13.7695256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119002 2022-11-23T03:18:13.7695877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7696318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7696903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7697360Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7697786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7698417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7698853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7699436Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7699951Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7700387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7701048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7701727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7702292Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7702853Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7703385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7703936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7704347Z dist init r=1, world=2 2022-11-23T03:18:13.7705799Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7706718Z warnings.warn( 2022-11-23T03:18:13.7707680Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7708362Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7708685Z dist init r=0, world=2 2022-11-23T03:18:13.7710137Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7711053Z warnings.warn( 2022-11-23T03:18:13.7712016Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7712705Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7713016Z ok (6.534s) 2022-11-23T03:18:13.7713706Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119150 2022-11-23T03:18:13.7714464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119151 2022-11-23T03:18:13.7715216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7715729Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7716429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7716976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7717499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7718252Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7719014Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7719720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7720253Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7720769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7721427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7722112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7722673Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7723167Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7723616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7724068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7724412Z dist init r=0, world=2 2022-11-23T03:18:13.7725611Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7726366Z warnings.warn( 2022-11-23T03:18:13.7727167Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7727744Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7728012Z dist init r=1, world=2 2022-11-23T03:18:13.7729208Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7729960Z warnings.warn( 2022-11-23T03:18:13.7730791Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7731363Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7731679Z ok (6.630s) 2022-11-23T03:18:13.7732208Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119303 2022-11-23T03:18:13.7732830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119304 2022-11-23T03:18:13.7733457Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7733891Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7734473Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7734932Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7735368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7736047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7736478Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7737058Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7737513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7737945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7738601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7739291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7739852Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7740355Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7740792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7741251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7741594Z dist init r=1, world=2 2022-11-23T03:18:13.7742791Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7743546Z warnings.warn( 2022-11-23T03:18:13.7744340Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7744905Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7745164Z dist init r=0, world=2 2022-11-23T03:18:13.7746360Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7747113Z warnings.warn( 2022-11-23T03:18:13.7747963Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7748539Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7748797Z ok (6.535s) 2022-11-23T03:18:13.7749317Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119452 2022-11-23T03:18:13.7749936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119453 2022-11-23T03:18:13.7750556Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7750992Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7751567Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7752077Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7752513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7753146Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7753583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7754165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7754620Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7755047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7755703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7756395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7756962Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7757462Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7757902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7758364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7758694Z dist init r=1, world=2 2022-11-23T03:18:13.7758931Z dist init r=0, world=2 2022-11-23T03:18:13.7760143Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7760916Z warnings.warn( 2022-11-23T03:18:13.7761715Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7762288Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7763513Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7764274Z warnings.warn( 2022-11-23T03:18:13.7765120Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7765691Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7765948Z ok (6.634s) 2022-11-23T03:18:13.7766356Z test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119605 2022-11-23T03:18:13.7766871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119606 2022-11-23T03:18:13.7767489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7767926Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7768510Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7769015Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7769451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7770074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7770510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7771224Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7771681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7772122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7772774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7773469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7774033Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7774522Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7774963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7775422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7775763Z dist init r=1, world=2 2022-11-23T03:18:13.7776001Z dist init r=0, world=2 2022-11-23T03:18:13.7776230Z ok (4.526s) 2022-11-23T03:18:13.7776725Z test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119752 2022-11-23T03:18:13.7777332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119753 2022-11-23T03:18:13.7777952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7778387Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7778968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7779426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7779862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7780488Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7780914Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7781561Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7782022Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7782454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7783108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7783797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7784362Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7784852Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7785290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7785749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7786155Z dist init r=1, world=2 2022-11-23T03:18:13.7787369Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7788125Z warnings.warn( 2022-11-23T03:18:13.7788361Z dist init r=0, world=2 2022-11-23T03:18:13.7789567Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7790323Z warnings.warn( 2022-11-23T03:18:13.7790547Z ok (4.529s) 2022-11-23T03:18:13.7791037Z test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119895 2022-11-23T03:18:13.7791633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119896 2022-11-23T03:18:13.7792250Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7792684Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7793264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7793720Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7794163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7794793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7795218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7795797Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7796252Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7796686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7797341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7798090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7798666Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7799151Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7799591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7800051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7800392Z dist init r=0, world=2 2022-11-23T03:18:13.7801595Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7802425Z warnings.warn( 2022-11-23T03:18:13.7802663Z dist init r=1, world=2 2022-11-23T03:18:13.7803868Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7804623Z warnings.warn( 2022-11-23T03:18:13.7804847Z ok (4.430s) 2022-11-23T03:18:13.7805341Z test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120038 2022-11-23T03:18:13.7805940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120039 2022-11-23T03:18:13.7806563Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7807002Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7807580Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7808036Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7808471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7809090Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7809528Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7810104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7810626Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7811061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7811714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7812405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7812970Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7813461Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7813898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7814357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7814698Z dist init r=1, world=2 2022-11-23T03:18:13.7815970Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7816735Z warnings.warn( 2022-11-23T03:18:13.7816968Z dist init r=0, world=2 2022-11-23T03:18:13.7818169Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7818968Z warnings.warn( 2022-11-23T03:18:13.7819182Z ok (4.333s) 2022-11-23T03:18:13.7819683Z test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120181 2022-11-23T03:18:13.7820280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120182 2022-11-23T03:18:13.7820904Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7821342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7821922Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7822376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7822812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7823429Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7823865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7824442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7824897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7825327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7825978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7826662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7827239Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7827725Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7828166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7828699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7829112Z dist init r=1, world=2 2022-11-23T03:18:13.7830560Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7831477Z warnings.warn( 2022-11-23T03:18:13.7831761Z dist init r=0, world=2 2022-11-23T03:18:13.7833293Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7834203Z warnings.warn( 2022-11-23T03:18:13.7834463Z ok (4.829s) 2022-11-23T03:18:13.7835082Z test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120324 2022-11-23T03:18:13.7835799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120325 2022-11-23T03:18:13.7836552Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7837143Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7837847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7838394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7838922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7839672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7840201Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7840834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7841291Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7841737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7842391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7843082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7843639Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7844140Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7844577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7845039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7845381Z dist init r=1, world=2 2022-11-23T03:18:13.7846584Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7847353Z warnings.warn( 2022-11-23T03:18:13.7847588Z dist init r=0, world=2 2022-11-23T03:18:13.7848785Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7849542Z warnings.warn( 2022-11-23T03:18:13.7849814Z ok (4.528s) 2022-11-23T03:18:13.7850318Z test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120467 2022-11-23T03:18:13.7850964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120468 2022-11-23T03:18:13.7851587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7852022Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7852605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7853061Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7853488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7854190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7854624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7855205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7855660Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7856093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7856751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7857436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7857991Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7858498Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7858937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7859398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7859739Z dist init r=1, world=2 2022-11-23T03:18:13.7860940Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7861697Z warnings.warn( 2022-11-23T03:18:13.7861936Z dist init r=0, world=2 2022-11-23T03:18:13.7863133Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7863892Z warnings.warn( 2022-11-23T03:18:13.7864108Z ok (4.536s) 2022-11-23T03:18:13.7864609Z test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120610 2022-11-23T03:18:13.7865205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120611 2022-11-23T03:18:13.7865821Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7866323Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7866914Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7867373Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7867799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7868431Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7868865Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7869447Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7869904Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7870342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7871053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7871729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7872298Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7872795Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7873229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7873690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7874035Z dist init r=1, world=2 2022-11-23T03:18:13.7875239Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7875997Z warnings.warn( 2022-11-23T03:18:13.7876232Z dist init r=0, world=2 2022-11-23T03:18:13.7877420Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7878168Z warnings.warn( 2022-11-23T03:18:13.7878392Z ok (4.828s) 2022-11-23T03:18:13.7878901Z test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120753 2022-11-23T03:18:13.7879496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120754 2022-11-23T03:18:13.7880111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7880548Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7881135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7881593Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7882019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7882705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7883150Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7883734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7884189Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7884623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7885285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7885966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7886533Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7887033Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7887529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7887992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7888333Z dist init r=1, world=2 2022-11-23T03:18:13.7889537Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7890299Z warnings.warn( 2022-11-23T03:18:13.7890631Z dist init r=0, world=2 2022-11-23T03:18:13.7891839Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7892597Z warnings.warn( 2022-11-23T03:18:13.7892824Z ok (4.528s) 2022-11-23T03:18:13.7893288Z test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120896 2022-11-23T03:18:13.7893851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120897 2022-11-23T03:18:13.7894472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7894906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7895494Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7895941Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7896373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7897001Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7897435Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7898011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7898465Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7898897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7899616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7900316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7900886Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7901386Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7901822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7902281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7902625Z dist init r=0, world=2 2022-11-23T03:18:13.7902857Z dist init r=1, world=2 2022-11-23T03:18:13.7903087Z ok (5.028s) 2022-11-23T03:18:13.7903547Z test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121039 2022-11-23T03:18:13.7904171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121040 2022-11-23T03:18:13.7904788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7905223Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7905806Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7906257Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7906694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7907323Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7907760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7908350Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7908806Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7909241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7909897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7910576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7911140Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7911641Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7912079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7912542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7912643Z dist init r=0, world=2 2022-11-23T03:18:13.7912739Z dist init r=1, world=2 2022-11-23T03:18:13.7912825Z ok (4.426s) 2022-11-23T03:18:13.7913134Z test_summon_full_param_writeback (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121182 2022-11-23T03:18:13.7913343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121183 2022-11-23T03:18:13.7913720Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7913874Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7914260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7914440Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7914727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7915109Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7915272Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7915657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7915833Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7916057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7916455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7916849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7917179Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7917454Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7917670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7917891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7917986Z dist init r=1, world=2 2022-11-23T03:18:13.7918081Z dist init r=0, world=2 2022-11-23T03:18:13.7918165Z ok (4.625s) 2022-11-23T03:18:13.7918522Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121325 2022-11-23T03:18:13.7918727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121326 2022-11-23T03:18:13.7919093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7919263Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7919650Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7919830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7920053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7920433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7920593Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7920977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7921152Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7921381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7921780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7922174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7922450Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7922723Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7922939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7923155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7923254Z dist init r=1, world=2 2022-11-23T03:18:13.7923349Z dist init r=0, world=2 2022-11-23T03:18:13.7923435Z ok (4.627s) 2022-11-23T03:18:13.7923856Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121472 2022-11-23T03:18:13.7924063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121473 2022-11-23T03:18:13.7924432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7924596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7924979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7925157Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7925381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7925755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7925966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7926353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7926528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7926752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7927150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7927546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7927821Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7928094Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7928314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7928529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7928624Z dist init r=0, world=2 2022-11-23T03:18:13.7929497Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7929591Z warnings.warn( 2022-11-23T03:18:13.7930457Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7930704Z warnings.warn( 2022-11-23T03:18:13.7930802Z dist init r=1, world=2 2022-11-23T03:18:13.7931672Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:818: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7931768Z warnings.warn( 2022-11-23T03:18:13.7932619Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-11-23T03:18:13.7932718Z warnings.warn( 2022-11-23T03:18:13.7932860Z ok (4.428s) 2022-11-23T03:18:13.7933219Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121615 2022-11-23T03:18:13.7933417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121616 2022-11-23T03:18:13.7933795Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7933958Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7934346Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7934521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7934747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7935175Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7935338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7935724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7935899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7936123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7936523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7936917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7937192Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7937472Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7937687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7937902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7937999Z dist init r=1, world=2 2022-11-23T03:18:13.7938095Z dist init r=0, world=2 2022-11-23T03:18:13.7938180Z ok (5.029s) 2022-11-23T03:18:13.7938523Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121760 2022-11-23T03:18:13.7938729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121761 2022-11-23T03:18:13.7939105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7939276Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7939666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7939843Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7940068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7940441Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7940601Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7940988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7941162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7941387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7941844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7942245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7942519Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7942792Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7943008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7943224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7943323Z dist init r=0, world=2 2022-11-23T03:18:13.7943419Z dist init r=1, world=2 2022-11-23T03:18:13.7943504Z ok (4.827s) 2022-11-23T03:18:13.7943833Z test_summon_full_params_respects_reshard_after_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121903 2022-11-23T03:18:13.7944103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121904 2022-11-23T03:18:13.7944483Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7944645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7945030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7945210Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7945435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7945809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7945974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7946364Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7946542Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7946765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7947164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7947560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7947834Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7948107Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7948322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7948543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7948640Z dist init r=0, world=2 2022-11-23T03:18:13.7948736Z dist init r=1, world=2 2022-11-23T03:18:13.7949794Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7949891Z warnings.warn( 2022-11-23T03:18:13.7950549Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7950728Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7951793Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7951889Z warnings.warn( 2022-11-23T03:18:13.7952547Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:18:13.7952675Z warnings.warn(message, UserWarning) 2022-11-23T03:18:13.7952763Z ok (6.528s) 2022-11-23T03:18:13.7953097Z test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122052 2022-11-23T03:18:13.7953305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122053 2022-11-23T03:18:13.7953683Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7953846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7954258Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7954468Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7954739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7955188Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7955390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7955857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7956064Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7956334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7956817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7957299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7957630Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7957960Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7958227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7958486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7958603Z dist init r=1, world=2 2022-11-23T03:18:13.7959885Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7960000Z warnings.warn( 2022-11-23T03:18:13.7960116Z dist init r=0, world=2 2022-11-23T03:18:13.7961441Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:18:13.7961557Z warnings.warn( 2022-11-23T03:18:13.7961663Z ok (4.422s) 2022-11-23T03:18:13.7961839Z test_with_grads_core (__main__.TestSummonFullParams) 2022-11-23T03:18:13.7962200Z Tests the core usage of ``summon_full_params(with_grads=True)``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122195 2022-11-23T03:18:13.7962451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122196 2022-11-23T03:18:13.7962906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7963107Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7963631Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7963844Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7964116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7964570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7964763Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7965227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7965438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7965707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7966191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7966671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7967007Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7967334Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7967593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7967858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7968121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7968384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7968644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7968900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7969160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7969421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7969688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7969948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7970206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7970462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7970884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7971118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7971405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7971622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7971838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7972053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7972268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7972482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7972698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7972912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7973171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7973390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7973604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7973820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:18:13.7973907Z dist init r=1, world=2 2022-11-23T03:18:13.7974006Z dist init r=0, world=2 2022-11-23T03:18:13.7974092Z ok (9.732s) 2022-11-23T03:18:13.7974262Z test_with_grads_none_grads (__main__.TestSummonFullParams) 2022-11-23T03:18:13.7974745Z Tests that if all ranks' ``FlatParameter`` has ``None`` gradient, then ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122348 2022-11-23T03:18:13.7974953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122349 2022-11-23T03:18:13.7975331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7975499Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7975885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7976060Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7976287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7976656Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7976817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7977202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7977378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7977608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:18:13.7978011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7978409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:18:13.7978684Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7978957Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7979163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7979380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:18:13.7979477Z dist init r=0, world=2 2022-11-23T03:18:13.7979572Z dist init r=1, world=2 2022-11-23T03:18:13.7979658Z ok (4.531s) 2022-11-23T03:18:13.7980038Z test_summon_full_param_writeback (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122491 2022-11-23T03:18:13.7980422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:18:13.7980583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:18:13.7980970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:18:13.7981147Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:18:13.7981373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:18:13.7981771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:18:13.7982046Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:18:13.7982313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:18:13.7982411Z dist init r=0, world=1 2022-11-23T03:18:13.7982496Z ok (4.418s) 2022-11-23T03:18:13.7982504Z 2022-11-23T03:18:13.7982776Z ---------------------------------------------------------------------- 2022-11-23T03:18:13.7982880Z Ran 52 tests in 266.240s 2022-11-23T03:18:13.7982886Z 2022-11-23T03:18:13.7982955Z OK 2022-11-23T03:18:13.7982971Z 2022-11-23T03:18:13.7983071Z Generating XML reports... 2022-11-23T03:18:13.7983554Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221123031345.xml 2022-11-23T03:18:13.7984053Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221123031345.xml 2022-11-23T03:18:13.7984059Z 2022-11-23T03:18:13.7984457Z ##[endgroup] 2022-11-23T03:18:13.7984993Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_iej_4uxc) 2022-11-23T03:18:13.7985000Z 2022-11-23T03:18:13.7985301Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-11-23 03:18:13.725187] 2022-11-23T03:18:13.7985818Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:18:13.725821] 2022-11-23T03:19:12.7273519Z 2022-11-23T03:19:12.7274902Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_sharded_grad_scaler 2022-11-23T03:19:12.7277427Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_bmc7tj14) 2022-11-23T03:19:12.7281479Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler 2022-11-23T03:19:12.7282789Z 2022-11-23T03:19:12.7283166Z Running tests... 2022-11-23T03:19:12.7284876Z ---------------------------------------------------------------------- 2022-11-23T03:19:12.7287044Z test_grad_scaling (__main__.TestShardGradScaler) ... ok (0.658s) 2022-11-23T03:19:12.7289057Z test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... ok (0.002s) 2022-11-23T03:19:12.7291260Z test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... ok (0.011s) 2022-11-23T03:19:12.7294265Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122661 2022-11-23T03:19:12.7297868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122662 2022-11-23T03:19:12.7300988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7303866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7306927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7309327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7311284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:12.7314169Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7316420Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7318985Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7321194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7323195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:12.7327467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7331513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7334511Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7337530Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7340182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:12.7342286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:12.7345021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7347257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7355347Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7361146Z warnings.warn( 2022-11-23T03:19:12.7362882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7366730Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7368797Z warnings.warn( 2022-11-23T03:19:12.7369739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7371050Z dist init r=1, world=2 2022-11-23T03:19:12.7371702Z dist init r=0, world=2 2022-11-23T03:19:12.7372481Z ok (7.347s) 2022-11-23T03:19:12.7374534Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122814 2022-11-23T03:19:12.7377562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122815 2022-11-23T03:19:12.7380576Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7381850Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7383762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7385073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7386182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:12.7387311Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7387838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7388532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7389085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7389610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:12.7390521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7391360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7392052Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7392658Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7393191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:12.7393743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:12.7394307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7394872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7396469Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7397395Z warnings.warn( 2022-11-23T03:19:12.7397824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7399406Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7400300Z warnings.warn( 2022-11-23T03:19:12.7400655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7400996Z dist init r=0, world=2 2022-11-23T03:19:12.7401223Z dist init r=1, world=2 2022-11-23T03:19:12.7401451Z ok (6.635s) 2022-11-23T03:19:12.7401979Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122967 2022-11-23T03:19:12.7402604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122968 2022-11-23T03:19:12.7403222Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7403660Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7404298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7404756Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7405192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:12.7405825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7406257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7406840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7407295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7407731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:12.7408384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7409136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7409706Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7410203Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7410718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:12.7411188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:12.7411686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7412244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7413842Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7414767Z warnings.warn( 2022-11-23T03:19:12.7415196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7416772Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7417683Z warnings.warn( 2022-11-23T03:19:12.7418111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7418526Z dist init r=1, world=2 2022-11-23T03:19:12.7418807Z dist init r=0, world=2 2022-11-23T03:19:12.7419075Z ok (6.737s) 2022-11-23T03:19:12.7419692Z test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123120 2022-11-23T03:19:12.7420428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123121 2022-11-23T03:19:12.7421174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7421695Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7422396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7423031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7423561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:12.7424330Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7424848Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7425547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7426090Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7426614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:12.7427396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7428235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7428983Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7429571Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7430110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:12.7430566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:12.7431040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7431505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7432830Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7433594Z warnings.warn( 2022-11-23T03:19:12.7433951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7435256Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7436009Z warnings.warn( 2022-11-23T03:19:12.7436354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7436706Z dist init r=0, world=2 2022-11-23T03:19:12.7436940Z dist init r=1, world=2 2022-11-23T03:19:12.7437349Z ok (6.733s) 2022-11-23T03:19:12.7437868Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123273 2022-11-23T03:19:12.7438482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123274 2022-11-23T03:19:12.7439101Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7439534Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7440104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7440566Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7441062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:12.7441699Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7442130Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7442707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7443161Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7443582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:12.7444235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7444922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7445547Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7446043Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7446478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:12.7446939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:12.7447395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7447861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7449180Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7449945Z warnings.warn( 2022-11-23T03:19:12.7450198Z File "", line 1, in 2022-11-23T03:19:12.7450620Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:19:12.7450982Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:19:12.7451343Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:19:12.7451738Z return self._bootstrap(parent_sentinel) 2022-11-23T03:19:12.7452191Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:19:12.7452572Z self.run() 2022-11-23T03:19:12.7452959Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:19:12.7453386Z self._target(*self._args, **self._kwargs) 2022-11-23T03:19:12.7454027Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:19:12.7454483Z self.run_test(test_name, pipe) 2022-11-23T03:19:12.7455121Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:19:12.7455583Z getattr(self, test_name)() 2022-11-23T03:19:12.7456223Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:19:12.7456654Z fn() 2022-11-23T03:19:12.7457252Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:19:12.7457716Z test(self, **param_kwargs) 2022-11-23T03:19:12.7458338Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:19:12.7458794Z return func(*args, **kwargs) 2022-11-23T03:19:12.7459389Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2022-11-23T03:19:12.7459882Z self._test_fsdp_parity( 2022-11-23T03:19:12.7460529Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:19:12.7461024Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:19:12.7461709Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:19:12.7462162Z output = model(*input) 2022-11-23T03:19:12.7462755Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:19:12.7463211Z return forward_call(*input, **kwargs) 2022-11-23T03:19:12.7463883Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:19:12.7464420Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:19:12.7465193Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:19:12.7465651Z _lazy_init(state, module) 2022-11-23T03:19:12.7466262Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:19:12.7466733Z handle.init_flat_param_attributes() 2022-11-23T03:19:12.7467359Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:19:12.7467805Z return func(*args, **kwargs) 2022-11-23T03:19:12.7468461Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:19:12.7468908Z p_assert( 2022-11-23T03:19:12.7469470Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:19:12.7469916Z traceback.print_stack() 2022-11-23T03:19:12.7470339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7471654Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7472408Z warnings.warn( 2022-11-23T03:19:12.7472661Z File "", line 1, in 2022-11-23T03:19:12.7473019Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:19:12.7473376Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:19:12.7473731Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:19:12.7474097Z return self._bootstrap(parent_sentinel) 2022-11-23T03:19:12.7474475Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:19:12.7474794Z self.run() 2022-11-23T03:19:12.7475113Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:19:12.7475469Z self._target(*self._args, **self._kwargs) 2022-11-23T03:19:12.7475997Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:19:12.7476366Z self.run_test(test_name, pipe) 2022-11-23T03:19:12.7476900Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:19:12.7477282Z getattr(self, test_name)() 2022-11-23T03:19:12.7477809Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:19:12.7478164Z fn() 2022-11-23T03:19:12.7478722Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:19:12.7479100Z test(self, **param_kwargs) 2022-11-23T03:19:12.7479625Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:19:12.7479999Z return func(*args, **kwargs) 2022-11-23T03:19:12.7480427Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2022-11-23T03:19:12.7480832Z self._test_fsdp_parity( 2022-11-23T03:19:12.7481363Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:19:12.7481774Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:19:12.7482332Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:19:12.7482765Z output = model(*input) 2022-11-23T03:19:12.7483256Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:19:12.7483631Z return forward_call(*input, **kwargs) 2022-11-23T03:19:12.7484185Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:19:12.7484625Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:19:12.7485206Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:19:12.7485581Z _lazy_init(state, module) 2022-11-23T03:19:12.7486094Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:19:12.7486486Z handle.init_flat_param_attributes() 2022-11-23T03:19:12.7487009Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:19:12.7487383Z return func(*args, **kwargs) 2022-11-23T03:19:12.7487930Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:19:12.7488299Z p_assert( 2022-11-23T03:19:12.7488762Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:19:12.7489131Z traceback.print_stack() 2022-11-23T03:19:12.7489504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7489967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7490432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7490919Z dist init r=1, world=2 2022-11-23T03:19:12.7491376Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:19:12.7491898Z dist init r=0, world=2 2022-11-23T03:19:12.7492444Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:19:12.7492950Z ok (6.834s) 2022-11-23T03:19:12.7493560Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123426 2022-11-23T03:19:12.7494282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123427 2022-11-23T03:19:12.7495035Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7495562Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7496254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7496998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7497529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:12.7498292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7498817Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7499518Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7500067Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7500581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:12.7501373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7502280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7502965Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7503561Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7504085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:12.7504636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:12.7505192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7505743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7507323Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7508255Z warnings.warn( 2022-11-23T03:19:12.7508555Z File "", line 1, in 2022-11-23T03:19:12.7508983Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:19:12.7509409Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:19:12.7509836Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:19:12.7510279Z return self._bootstrap(parent_sentinel) 2022-11-23T03:19:12.7510649Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:19:12.7510968Z self.run() 2022-11-23T03:19:12.7511295Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:19:12.7511653Z self._target(*self._args, **self._kwargs) 2022-11-23T03:19:12.7512177Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:19:12.7512552Z self.run_test(test_name, pipe) 2022-11-23T03:19:12.7513117Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:19:12.7513570Z getattr(self, test_name)() 2022-11-23T03:19:12.7514207Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:19:12.7514631Z fn() 2022-11-23T03:19:12.7515231Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:19:12.7515692Z test(self, **param_kwargs) 2022-11-23T03:19:12.7516309Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:19:12.7516846Z return func(*args, **kwargs) 2022-11-23T03:19:12.7517365Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2022-11-23T03:19:12.7517852Z self._test_fsdp_parity( 2022-11-23T03:19:12.7518495Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:19:12.7518985Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:19:12.7519669Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:19:12.7520118Z output = model(*input) 2022-11-23T03:19:12.7520707Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:19:12.7521165Z return forward_call(*input, **kwargs) 2022-11-23T03:19:12.7521831Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:19:12.7522433Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:19:12.7523124Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:19:12.7523582Z _lazy_init(state, module) 2022-11-23T03:19:12.7524191Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:19:12.7524662Z handle.init_flat_param_attributes() 2022-11-23T03:19:12.7525289Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:19:12.7525732Z return func(*args, **kwargs) 2022-11-23T03:19:12.7526388Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:19:12.7526832Z p_assert( 2022-11-23T03:19:12.7527407Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:19:12.7527842Z traceback.print_stack() 2022-11-23T03:19:12.7528291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7529885Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7530755Z warnings.warn( 2022-11-23T03:19:12.7531005Z File "", line 1, in 2022-11-23T03:19:12.7531362Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:19:12.7531719Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:19:12.7532124Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:19:12.7532548Z return self._bootstrap(parent_sentinel) 2022-11-23T03:19:12.7533004Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:19:12.7533390Z self.run() 2022-11-23T03:19:12.7533777Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:19:12.7534204Z self._target(*self._args, **self._kwargs) 2022-11-23T03:19:12.7534829Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:19:12.7535278Z self.run_test(test_name, pipe) 2022-11-23T03:19:12.7535921Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:19:12.7536371Z getattr(self, test_name)() 2022-11-23T03:19:12.7537073Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:19:12.7537507Z fn() 2022-11-23T03:19:12.7538114Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:19:12.7538569Z test(self, **param_kwargs) 2022-11-23T03:19:12.7539189Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:19:12.7539648Z return func(*args, **kwargs) 2022-11-23T03:19:12.7540168Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2022-11-23T03:19:12.7540652Z self._test_fsdp_parity( 2022-11-23T03:19:12.7541290Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:19:12.7541784Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:19:12.7542459Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:19:12.7542985Z output = model(*input) 2022-11-23T03:19:12.7543572Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:19:12.7544029Z return forward_call(*input, **kwargs) 2022-11-23T03:19:12.7544703Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:19:12.7545234Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:19:12.7545924Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:19:12.7546376Z _lazy_init(state, module) 2022-11-23T03:19:12.7546991Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:19:12.7547459Z handle.init_flat_param_attributes() 2022-11-23T03:19:12.7548094Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:19:12.7548537Z return func(*args, **kwargs) 2022-11-23T03:19:12.7549188Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:19:12.7549630Z p_assert( 2022-11-23T03:19:12.7550186Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:19:12.7550551Z traceback.print_stack() 2022-11-23T03:19:12.7550923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7551392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7551859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7552204Z dist init r=0, world=2 2022-11-23T03:19:12.7552659Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:19:12.7553092Z dist init r=1, world=2 2022-11-23T03:19:12.7553550Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:19:12.7553968Z ok (6.735s) 2022-11-23T03:19:12.7554496Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123579 2022-11-23T03:19:12.7555118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123580 2022-11-23T03:19:12.7555739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7556177Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7556816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7557281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7557717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:12.7558353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7558789Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7559365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7559820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7560254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:12.7560968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7561656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7562221Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7562723Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7563161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:12.7563623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:12.7564087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7564546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7565858Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7566625Z warnings.warn( 2022-11-23T03:19:12.7566878Z File "", line 1, in 2022-11-23T03:19:12.7567234Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:19:12.7567592Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:19:12.7567952Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:19:12.7568312Z return self._bootstrap(parent_sentinel) 2022-11-23T03:19:12.7568679Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:19:12.7569002Z self.run() 2022-11-23T03:19:12.7569326Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:19:12.7569681Z self._target(*self._args, **self._kwargs) 2022-11-23T03:19:12.7570201Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:19:12.7570638Z self.run_test(test_name, pipe) 2022-11-23T03:19:12.7571176Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:19:12.7571560Z getattr(self, test_name)() 2022-11-23T03:19:12.7572083Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:19:12.7572438Z fn() 2022-11-23T03:19:12.7572938Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:19:12.7573323Z test(self, **param_kwargs) 2022-11-23T03:19:12.7573914Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:19:12.7574287Z return func(*args, **kwargs) 2022-11-23T03:19:12.7574716Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2022-11-23T03:19:12.7575116Z self._test_fsdp_parity( 2022-11-23T03:19:12.7575651Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:19:12.7576061Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:19:12.7576623Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:19:12.7577007Z output = model(*input) 2022-11-23T03:19:12.7577488Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:19:12.7577923Z return forward_call(*input, **kwargs) 2022-11-23T03:19:12.7578484Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:19:12.7578928Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:19:12.7579500Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:19:12.7579879Z _lazy_init(state, module) 2022-11-23T03:19:12.7580395Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:19:12.7580778Z handle.init_flat_param_attributes() 2022-11-23T03:19:12.7581303Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:19:12.7581672Z return func(*args, **kwargs) 2022-11-23T03:19:12.7582218Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:19:12.7582594Z p_assert( 2022-11-23T03:19:12.7583070Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:19:12.7583431Z traceback.print_stack() 2022-11-23T03:19:12.7583805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7585114Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7585869Z warnings.warn( 2022-11-23T03:19:12.7586123Z File "", line 1, in 2022-11-23T03:19:12.7586486Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:19:12.7586841Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:19:12.7587201Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:19:12.7587550Z return self._bootstrap(parent_sentinel) 2022-11-23T03:19:12.7587930Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:19:12.7588251Z self.run() 2022-11-23T03:19:12.7588575Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:19:12.7588932Z self._target(*self._args, **self._kwargs) 2022-11-23T03:19:12.7589452Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:19:12.7589817Z self.run_test(test_name, pipe) 2022-11-23T03:19:12.7590351Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:19:12.7590734Z getattr(self, test_name)() 2022-11-23T03:19:12.7591310Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:19:12.7591670Z fn() 2022-11-23T03:19:12.7592171Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:19:12.7592550Z test(self, **param_kwargs) 2022-11-23T03:19:12.7593066Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:19:12.7593451Z return func(*args, **kwargs) 2022-11-23T03:19:12.7593889Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2022-11-23T03:19:12.7594289Z self._test_fsdp_parity( 2022-11-23T03:19:12.7594820Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:19:12.7595284Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:19:12.7595854Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:19:12.7596228Z output = model(*input) 2022-11-23T03:19:12.7596712Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:19:12.7597089Z return forward_call(*input, **kwargs) 2022-11-23T03:19:12.7597645Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:19:12.7598088Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:19:12.7598657Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:19:12.7599039Z _lazy_init(state, module) 2022-11-23T03:19:12.7599547Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:19:12.7599942Z handle.init_flat_param_attributes() 2022-11-23T03:19:12.7600471Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:19:12.7600841Z return func(*args, **kwargs) 2022-11-23T03:19:12.7601392Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:19:12.7601763Z p_assert( 2022-11-23T03:19:12.7602226Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:19:12.7602591Z traceback.print_stack() 2022-11-23T03:19:12.7602968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7603439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7603905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7604256Z dist init r=1, world=2 2022-11-23T03:19:12.7604718Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:19:12.7605136Z dist init r=0, world=2 2022-11-23T03:19:12.7605589Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:19:12.7606007Z ok (6.738s) 2022-11-23T03:19:12.7606520Z test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123732 2022-11-23T03:19:12.7607131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123733 2022-11-23T03:19:12.7607755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7608260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7608849Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7609299Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7609739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:12.7610368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:12.7610947Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:12.7611535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:12.7611993Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:12.7612494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:12.7613144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7613837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:12.7614403Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7614902Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:12.7615340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:12.7615798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:12.7616263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7616720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7618042Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7618808Z warnings.warn( 2022-11-23T03:19:12.7619059Z File "", line 1, in 2022-11-23T03:19:12.7619417Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:19:12.7619775Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:19:12.7620133Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:19:12.7620491Z return self._bootstrap(parent_sentinel) 2022-11-23T03:19:12.7620865Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:19:12.7621190Z self.run() 2022-11-23T03:19:12.7621509Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:19:12.7621870Z self._target(*self._args, **self._kwargs) 2022-11-23T03:19:12.7622391Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:19:12.7622769Z self.run_test(test_name, pipe) 2022-11-23T03:19:12.7623311Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:19:12.7623684Z getattr(self, test_name)() 2022-11-23T03:19:12.7624206Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:19:12.7624559Z fn() 2022-11-23T03:19:12.7625128Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:19:12.7625518Z test(self, **param_kwargs) 2022-11-23T03:19:12.7626047Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:19:12.7626418Z return func(*args, **kwargs) 2022-11-23T03:19:12.7626847Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2022-11-23T03:19:12.7627248Z self._test_fsdp_parity( 2022-11-23T03:19:12.7627779Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:19:12.7628188Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:19:12.7628751Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:19:12.7629135Z output = model(*input) 2022-11-23T03:19:12.7629693Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:19:12.7630072Z return forward_call(*input, **kwargs) 2022-11-23T03:19:12.7630627Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:19:12.7631068Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:19:12.7631640Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:19:12.7632018Z _lazy_init(state, module) 2022-11-23T03:19:12.7632531Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:19:12.7632916Z handle.init_flat_param_attributes() 2022-11-23T03:19:12.7633438Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:19:12.7633807Z return func(*args, **kwargs) 2022-11-23T03:19:12.7634358Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:19:12.7634730Z p_assert( 2022-11-23T03:19:12.7635207Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:19:12.7635573Z traceback.print_stack() 2022-11-23T03:19:12.7635937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7637254Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:12.7638023Z warnings.warn( 2022-11-23T03:19:12.7638284Z File "", line 1, in 2022-11-23T03:19:12.7638669Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:19:12.7639096Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:19:12.7639526Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:19:12.7639949Z return self._bootstrap(parent_sentinel) 2022-11-23T03:19:12.7640407Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:19:12.7640795Z self.run() 2022-11-23T03:19:12.7641182Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:19:12.7641619Z self._target(*self._args, **self._kwargs) 2022-11-23T03:19:12.7642246Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:19:12.7642697Z self.run_test(test_name, pipe) 2022-11-23T03:19:12.7643416Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:19:12.7643881Z getattr(self, test_name)() 2022-11-23T03:19:12.7644515Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:19:12.7644935Z fn() 2022-11-23T03:19:12.7645533Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 247, in instantiated_test 2022-11-23T03:19:12.7646000Z test(self, **param_kwargs) 2022-11-23T03:19:12.7646614Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:19:12.7647079Z return func(*args, **kwargs) 2022-11-23T03:19:12.7647597Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py", line 171, in test_fsdp_ddp_parity_with_grad_scaler 2022-11-23T03:19:12.7648075Z self._test_fsdp_parity( 2022-11-23T03:19:12.7648803Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 1008, in _test_fsdp_parity 2022-11-23T03:19:12.7649294Z fsdp_loss = self._train_for_several_steps( 2022-11-23T03:19:12.7649973Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 827, in _train_for_several_steps 2022-11-23T03:19:12.7650437Z output = model(*input) 2022-11-23T03:19:12.7650980Z File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1427, in _call_impl 2022-11-23T03:19:12.7651358Z return forward_call(*input, **kwargs) 2022-11-23T03:19:12.7651909Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 685, in forward 2022-11-23T03:19:12.7652413Z args, kwargs = _root_pre_forward(self, self, *args, **kwargs) 2022-11-23T03:19:12.7653108Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 334, in _root_pre_forward 2022-11-23T03:19:12.7653576Z _lazy_init(state, module) 2022-11-23T03:19:12.7654184Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 62, in _lazy_init 2022-11-23T03:19:12.7654658Z handle.init_flat_param_attributes() 2022-11-23T03:19:12.7655286Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:19:12.7655730Z return func(*args, **kwargs) 2022-11-23T03:19:12.7656382Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/flat_param.py", line 711, in init_flat_param_attributes 2022-11-23T03:19:12.7656824Z p_assert( 2022-11-23T03:19:12.7657392Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:19:12.7657820Z traceback.print_stack() 2022-11-23T03:19:12.7658274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7658844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7659407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:19:12.7659826Z dist init r=0, world=2 2022-11-23T03:19:12.7660382Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:0 after the FSDP constructor. 2022-11-23T03:19:12.7660840Z dist init r=1, world=2 2022-11-23T03:19:12.7661289Z Expects the `FlatParameter` to be offloaded to CPU since CPU offloading is enabled. You may be accidentally moving the model to cuda:1 after the FSDP constructor. 2022-11-23T03:19:12.7661702Z ok (6.635s) 2022-11-23T03:19:12.7661837Z 2022-11-23T03:19:12.7662113Z ---------------------------------------------------------------------- 2022-11-23T03:19:12.7662430Z Ran 11 tests in 55.068s 2022-11-23T03:19:12.7662580Z 2022-11-23T03:19:12.7662657Z OK 2022-11-23T03:19:12.7662777Z 2022-11-23T03:19:12.7662890Z Generating XML reports... 2022-11-23T03:19:12.7663590Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221123031815.xml 2022-11-23T03:19:12.7664481Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221123031815.xml 2022-11-23T03:19:12.7664880Z 2022-11-23T03:19:12.7665282Z ##[endgroup] 2022-11-23T03:19:12.7665949Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_bmc7tj14) 2022-11-23T03:19:12.7666328Z 2022-11-23T03:19:12.7666601Z Running distributed/fsdp/test_fsdp_overlap ... [2022-11-23 03:19:12.730334] 2022-11-23T03:19:12.7667299Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:19:12.733300] 2022-11-23T03:19:17.3580725Z 2022-11-23T03:19:17.3581516Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_overlap 2022-11-23T03:19:17.3583040Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_overlap_f14lva4t) 2022-11-23T03:19:17.3584277Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap 2022-11-23T03:19:17.3584608Z 2022-11-23T03:19:17.3584748Z Running tests... 2022-11-23T03:19:17.3585252Z ---------------------------------------------------------------------- 2022-11-23T03:19:17.3586649Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79017 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.650s) 2022-11-23T03:19:17.3588476Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s) 2022-11-23T03:19:17.3589197Z 2022-11-23T03:19:17.3589513Z ---------------------------------------------------------------------- 2022-11-23T03:19:17.3589887Z Ran 2 tests in 0.651s 2022-11-23T03:19:17.3590067Z 2022-11-23T03:19:17.3590174Z OK (skipped=2) 2022-11-23T03:19:17.3590343Z 2022-11-23T03:19:17.3590463Z Generating XML reports... 2022-11-23T03:19:17.3591236Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221123031914.xml 2022-11-23T03:19:17.3592258Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221123031914.xml 2022-11-23T03:19:17.3592701Z 2022-11-23T03:19:17.3593031Z ##[endgroup] 2022-11-23T03:19:17.3593760Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_overlap_f14lva4t) 2022-11-23T03:19:17.3594168Z 2022-11-23T03:19:17.3594539Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-11-23 03:19:17.358261] 2022-11-23T03:19:17.3595418Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:19:17.358914] 2022-11-23T03:19:28.8401426Z 2022-11-23T03:19:28.8402882Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping 2022-11-23T03:19:28.8405821Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_qg45jd59) 2022-11-23T03:19:28.8409032Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping 2022-11-23T03:19:28.8409853Z 2022-11-23T03:19:28.8410180Z Running tests... 2022-11-23T03:19:28.8411620Z ---------------------------------------------------------------------- 2022-11-23T03:19:28.8412953Z test_multiple_wrapping (__main__.TestMultipleWrapping) 2022-11-23T03:19:28.8414573Z This test simulates wrapping the module after training to run inference. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124018 2022-11-23T03:19:28.8416156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124019 2022-11-23T03:19:28.8417861Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:28.8419033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:28.8420590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:28.8422127Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:28.8423269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:19:28.8424974Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:19:28.8426128Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:19:28.8427662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:19:28.8428865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:19:28.8430012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:19:28.8431763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:28.8433622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:19:28.8435114Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:28.8436442Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:19:28.8437604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:19:28.8438829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:19:28.8439743Z dist init r=0, world=2 2022-11-23T03:19:28.8442990Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:28.8445044Z warnings.warn( 2022-11-23T03:19:28.8445659Z dist init r=1, world=2 2022-11-23T03:19:28.8448880Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:19:28.8451026Z warnings.warn( 2022-11-23T03:19:28.8451593Z ok (7.481s) 2022-11-23T03:19:28.8451945Z 2022-11-23T03:19:28.8452662Z ---------------------------------------------------------------------- 2022-11-23T03:19:28.8453479Z Ran 1 test in 7.481s 2022-11-23T03:19:28.8453866Z 2022-11-23T03:19:28.8454073Z OK 2022-11-23T03:19:28.8454381Z 2022-11-23T03:19:28.8454842Z Generating XML reports... 2022-11-23T03:19:28.8456550Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221123031919.xml 2022-11-23T03:19:28.8457495Z 2022-11-23T03:19:28.8458247Z ##[endgroup] 2022-11-23T03:19:28.8459966Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_qg45jd59) 2022-11-23T03:19:28.8460951Z 2022-11-23T03:19:28.8461719Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-11-23 03:19:28.840256] 2022-11-23T03:19:28.8463661Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:19:28.840915] 2022-11-23T03:25:03.5996557Z 2022-11-23T03:25:03.6002090Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision 2022-11-23T03:25:03.6004812Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_nkrqo773) 2022-11-23T03:25:03.6007441Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision 2022-11-23T03:25:03.6008380Z 2022-11-23T03:25:03.6008716Z Running tests... 2022-11-23T03:25:03.6011299Z ---------------------------------------------------------------------- 2022-11-23T03:25:03.6013064Z test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124238 2022-11-23T03:25:03.6015323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124239 2022-11-23T03:25:03.6017602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6019138Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6021048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6022603Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6024124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6026256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6027682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6029608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6031197Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6032650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6034863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6037191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6039074Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6040757Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6043555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6045084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6046212Z dist init r=1, world=2 2022-11-23T03:25:03.6047004Z dist init r=0, world=2 2022-11-23T03:25:03.6047774Z ok (7.144s) 2022-11-23T03:25:03.6049935Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124391 2022-11-23T03:25:03.6052195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124392 2022-11-23T03:25:03.6054272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6055760Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6057769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6059289Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6060661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6062531Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6063941Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6065508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6066723Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6067876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6069635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6071476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6072964Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6074295Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6075465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6076710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6079490Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6083075Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6086521Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6089952Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6091795Z dist init r=0, world=2 2022-11-23T03:25:03.6094009Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6095553Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6096285Z dist init r=1, world=2 2022-11-23T03:25:03.6098625Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6100169Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6100877Z ok (7.533s) 2022-11-23T03:25:03.6102261Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124544 2022-11-23T03:25:03.6103913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124545 2022-11-23T03:25:03.6105658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6106879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6108519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6110072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6111276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6113053Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6114262Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6115903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6117177Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6118396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6120237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6122162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6123754Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6125145Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6126361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6127654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6130864Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6134516Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6138135Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6141108Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6141728Z dist init r=0, world=2 2022-11-23T03:25:03.6142602Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6143183Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6143442Z dist init r=1, world=2 2022-11-23T03:25:03.6144248Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6144821Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6145080Z ok (7.732s) 2022-11-23T03:25:03.6145605Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124697 2022-11-23T03:25:03.6146279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124698 2022-11-23T03:25:03.6146903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6147340Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6147913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6148374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6148809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6149437Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6149871Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6150453Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6150910Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6151345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6151997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6152691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6153260Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6153761Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6154208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6154684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6155721Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6157006Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6158290Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6159631Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6160240Z dist init r=0, world=2 2022-11-23T03:25:03.6161054Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6161620Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6161896Z dist init r=1, world=2 2022-11-23T03:25:03.6162709Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6163337Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6163604Z ok (7.834s) 2022-11-23T03:25:03.6164106Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124850 2022-11-23T03:25:03.6164708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124851 2022-11-23T03:25:03.6165338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6165772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6166362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6166829Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6167279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6167906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6168339Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6168920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6169368Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6169799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6170452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6171256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6171942Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6172532Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6173051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6173585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6174811Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6176439Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6177989Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6179523Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6180239Z dist init r=0, world=2 2022-11-23T03:25:03.6180527Z dist init r=1, world=2 2022-11-23T03:25:03.6181386Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6182007Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6182840Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6183404Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6183661Z ok (7.833s) 2022-11-23T03:25:03.6184186Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125003 2022-11-23T03:25:03.6184798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125004 2022-11-23T03:25:03.6185424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6185866Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6186450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6186908Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6187343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6187976Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6188404Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6188984Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6189443Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6189886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6190540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6191226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6191790Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6192288Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6192718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6193183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6194254Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6195537Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6196799Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6198072Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6198709Z dist init r=0, world=2 2022-11-23T03:25:03.6198951Z dist init r=1, world=2 2022-11-23T03:25:03.6199755Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6200327Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6201158Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6201723Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6201974Z ok (7.831s) 2022-11-23T03:25:03.6202471Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125156 2022-11-23T03:25:03.6203063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125157 2022-11-23T03:25:03.6203679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6204120Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6204704Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6205163Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6205602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6206228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6206664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6207243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6207697Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6208132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6208789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6209475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6210035Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6210613Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6211148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6211969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6213303Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6214844Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6216371Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6218023Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6218747Z dist init r=0, world=2 2022-11-23T03:25:03.6219722Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6220419Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6220751Z dist init r=1, world=2 2022-11-23T03:25:03.6221546Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6222127Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6222395Z ok (7.733s) 2022-11-23T03:25:03.6222927Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125309 2022-11-23T03:25:03.6223555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125310 2022-11-23T03:25:03.6224185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6224632Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6225229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6225682Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6226125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6226765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6227211Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6227799Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6228261Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6228701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6229407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6230122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6230696Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6231206Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6231648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6232117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6233147Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6234485Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6235761Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6237036Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6237648Z dist init r=1, world=2 2022-11-23T03:25:03.6237899Z dist init r=0, world=2 2022-11-23T03:25:03.6238695Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6239271Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6240105Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6240685Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6240951Z ok (8.236s) 2022-11-23T03:25:03.6241459Z test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125462 2022-11-23T03:25:03.6242063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125463 2022-11-23T03:25:03.6242689Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6243122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6243713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6244183Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6244626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6245266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6245709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6246355Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6246821Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6247245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6247909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6248602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6249170Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6249670Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6250111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6250779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6251940Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6253472Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6255005Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6256534Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6257259Z dist init r=1, world=2 2022-11-23T03:25:03.6258212Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6258911Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6259235Z dist init r=0, world=2 2022-11-23T03:25:03.6260198Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6260885Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6261192Z ok (7.635s) 2022-11-23T03:25:03.6261727Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125615 2022-11-23T03:25:03.6262353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125616 2022-11-23T03:25:03.6262964Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6263409Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6263997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6264527Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6264971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6265607Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6266045Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6266622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6267083Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6267524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6268185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6268936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6269507Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6270007Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6270450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6270902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6271928Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6273205Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6274475Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6275742Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6277014Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6278283Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6279555Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6280861Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6282141Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6283411Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6283997Z dist init r=1, world=2 2022-11-23T03:25:03.6284240Z dist init r=0, world=2 2022-11-23T03:25:03.6284513Z ok (7.633s) 2022-11-23T03:25:03.6285020Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125768 2022-11-23T03:25:03.6285616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125769 2022-11-23T03:25:03.6286243Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6286682Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6287256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6287714Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6288151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6288788Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6289226Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6289807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6290263Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6290738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6291402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6292088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6292655Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6293153Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6293599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6294059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6295079Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6296350Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6297679Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6298960Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6300232Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6301501Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6302819Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6304084Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6305351Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6306614Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6307208Z dist init r=0, world=2 2022-11-23T03:25:03.6307447Z dist init r=1, world=2 2022-11-23T03:25:03.6307666Z ok (7.635s) 2022-11-23T03:25:03.6308203Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125921 2022-11-23T03:25:03.6308832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125922 2022-11-23T03:25:03.6309448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6309886Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6310467Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6310926Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6311351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6311979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6312414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6313055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6313523Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6313956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6314613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6315309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6315870Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6316370Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6316810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6317267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6318344Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6319609Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6320874Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6322144Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6323410Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6324667Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6325933Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6327188Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6328448Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6329751Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6330351Z dist init r=1, world=2 2022-11-23T03:25:03.6330627Z dist init r=0, world=2 2022-11-23T03:25:03.6330858Z ok (7.632s) 2022-11-23T03:25:03.6331362Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126074 2022-11-23T03:25:03.6331947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126075 2022-11-23T03:25:03.6332570Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6333068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6333657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6334116Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6334552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6335181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6335621Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6336194Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6336653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6337086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6337743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6338431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6338998Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6339497Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6339924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6340387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6341404Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6342671Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6343936Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6345272Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6346537Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6347801Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6349068Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6350391Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6351652Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6352916Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6353511Z dist init r=0, world=2 2022-11-23T03:25:03.6353749Z dist init r=1, world=2 2022-11-23T03:25:03.6353978Z ok (7.631s) 2022-11-23T03:25:03.6354509Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126227 2022-11-23T03:25:03.6355128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126228 2022-11-23T03:25:03.6355750Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6356188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6356766Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6357223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6357665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6358302Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6358738Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6359321Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6359779Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6360210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6360869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6361619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6362200Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6362698Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6363136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6363600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6364624Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6365900Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6367224Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6368493Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6369759Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6371148Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6372409Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6373675Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6374933Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6376189Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6376778Z dist init r=0, world=2 2022-11-23T03:25:03.6377021Z dist init r=1, world=2 2022-11-23T03:25:03.6377239Z ok (8.133s) 2022-11-23T03:25:03.6377808Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126380 2022-11-23T03:25:03.6378407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126381 2022-11-23T03:25:03.6379030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6379469Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6380054Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6380517Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6380942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6381577Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6382070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6382655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6383110Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6383547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6384204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6384893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6385457Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6385955Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6386399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6386861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6387874Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6389140Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6390402Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6391668Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6392923Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6394238Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6395504Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6396768Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6398026Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6399337Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6399929Z dist init r=1, world=2 2022-11-23T03:25:03.6400168Z dist init r=0, world=2 2022-11-23T03:25:03.6400396Z ok (7.632s) 2022-11-23T03:25:03.6400928Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126533 2022-11-23T03:25:03.6401546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126534 2022-11-23T03:25:03.6402176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6402618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6403202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6403666Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6404103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6404735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6405173Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6405747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6406211Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6406645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6407297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6407989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6408556Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6409053Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6409485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6409944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6411092Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6412373Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6413641Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6414958Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6416224Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6417484Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6418749Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6420008Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6421270Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6422541Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6423131Z dist init r=1, world=2 2022-11-23T03:25:03.6423370Z dist init r=0, world=2 2022-11-23T03:25:03.6423595Z ok (7.834s) 2022-11-23T03:25:03.6424093Z test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126686 2022-11-23T03:25:03.6424690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126687 2022-11-23T03:25:03.6425308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6425748Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6426381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6426845Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6427286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6427923Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6428359Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6428940Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6429397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6429827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6430542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6431233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6431798Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6432297Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6432736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6433194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6434215Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6435490Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6436749Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6438021Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6439298Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6440552Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6441815Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6443124Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6444391Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6445646Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6446289Z dist init r=1, world=2 2022-11-23T03:25:03.6446526Z dist init r=0, world=2 2022-11-23T03:25:03.6446744Z ok (7.731s) 2022-11-23T03:25:03.6447294Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126839 2022-11-23T03:25:03.6447936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126840 2022-11-23T03:25:03.6448560Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6449000Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6449590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6450050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6450486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6451178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6451614Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6452198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6452659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6453098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6453756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6454444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6455014Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6455514Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6455953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6456416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6457441Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6458811Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6460093Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6461352Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6461947Z dist init r=1, world=2 2022-11-23T03:25:03.6462755Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6463394Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6463662Z dist init r=0, world=2 2022-11-23T03:25:03.6464457Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6465030Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6465287Z ok (7.732s) 2022-11-23T03:25:03.6465801Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126992 2022-11-23T03:25:03.6466411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126993 2022-11-23T03:25:03.6467031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6467472Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6468047Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6468505Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6468942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6469571Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6470006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6470587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6471044Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6471486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6472133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6472822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6473391Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6473891Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6474327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6474789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6475864Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6477140Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6478400Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6479677Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6480317Z dist init r=1, world=2 2022-11-23T03:25:03.6481124Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6481687Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6481958Z dist init r=0, world=2 2022-11-23T03:25:03.6482761Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6483331Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6483591Z ok (7.631s) 2022-11-23T03:25:03.6484142Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127145 2022-11-23T03:25:03.6484785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127146 2022-11-23T03:25:03.6485409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6485837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6486421Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6486881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6487318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6487954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6488390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6488973Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6489425Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6489861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6490629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6491331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6491898Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6492471Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6492913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6493375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6494399Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6495670Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6496997Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6498274Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6498860Z dist init r=0, world=2 2022-11-23T03:25:03.6499100Z dist init r=1, world=2 2022-11-23T03:25:03.6499906Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6500485Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6501314Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6501880Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6502137Z ok (7.934s) 2022-11-23T03:25:03.6502648Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127298 2022-11-23T03:25:03.6503251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127299 2022-11-23T03:25:03.6503868Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6504318Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6504902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6505356Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6505793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6506426Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6506863Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6507435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6507894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6508328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6509043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6509741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6510305Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6510802Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6511231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6511688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6512713Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6514037Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6515299Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6516570Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6517166Z dist init r=1, world=2 2022-11-23T03:25:03.6517969Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6518541Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6518810Z dist init r=0, world=2 2022-11-23T03:25:03.6519615Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6520183Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6520432Z ok (7.736s) 2022-11-23T03:25:03.6520981Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127451 2022-11-23T03:25:03.6521618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127452 2022-11-23T03:25:03.6522237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6522675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6523256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6523717Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6524156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6524776Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6525269Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6525854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6526310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6526742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6527400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6528094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6528661Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6529151Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6529646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6530106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6531186Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6532455Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6533726Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6534987Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6535588Z dist init r=0, world=2 2022-11-23T03:25:03.6536389Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6536958Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6537233Z dist init r=1, world=2 2022-11-23T03:25:03.6538051Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6538611Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6538867Z ok (8.035s) 2022-11-23T03:25:03.6539378Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127604 2022-11-23T03:25:03.6539987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127605 2022-11-23T03:25:03.6540604Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6541043Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6541692Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6542158Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6542586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6543220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6543657Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6544239Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6544695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6545133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6545788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6546524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6547094Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6547596Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6548031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6548500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6549520Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6550795Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6552053Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6553318Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6553914Z dist init r=1, world=2 2022-11-23T03:25:03.6554719Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6555284Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6555543Z dist init r=0, world=2 2022-11-23T03:25:03.6556348Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6556918Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6557179Z ok (7.834s) 2022-11-23T03:25:03.6557726Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127757 2022-11-23T03:25:03.6558424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127758 2022-11-23T03:25:03.6559049Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6559488Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6560061Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6560521Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6560962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6561590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6562023Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6562659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6563122Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6563914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6564565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6565256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6565823Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6566323Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6566761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6567235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6568255Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6569516Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6570845Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6572107Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6572707Z dist init r=0, world=2 2022-11-23T03:25:03.6573508Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6574074Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6574343Z dist init r=1, world=2 2022-11-23T03:25:03.6575217Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6575791Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6576050Z ok (7.834s) 2022-11-23T03:25:03.6576561Z test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127910 2022-11-23T03:25:03.6577170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127911 2022-11-23T03:25:03.6577798Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6578227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6578809Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6579323Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6579771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6580407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6580842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6581427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6581880Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6582320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6582972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6583664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6584239Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6584738Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6585177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6585641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6586653Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6587932Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6589203Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6590478Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6591068Z dist init r=0, world=2 2022-11-23T03:25:03.6591945Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6592523Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6592791Z dist init r=1, world=2 2022-11-23T03:25:03.6593598Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6594166Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6594422Z ok (7.835s) 2022-11-23T03:25:03.6594964Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128063 2022-11-23T03:25:03.6595590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128064 2022-11-23T03:25:03.6596272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6596708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6597293Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6597752Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6598190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6598823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6599268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6599839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6600305Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6600740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6601395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6602082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6602650Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6603149Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6603579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6604043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6605071Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6606348Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6607618Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6608939Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6610220Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6611617Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6612889Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6614233Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6615497Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6616764Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6617363Z dist init r=1, world=2 2022-11-23T03:25:03.6617603Z dist init r=0, world=2 2022-11-23T03:25:03.6617830Z ok (7.733s) 2022-11-23T03:25:03.6618342Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128216 2022-11-23T03:25:03.6618948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128217 2022-11-23T03:25:03.6619568Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6620006Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6620585Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6621047Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6621487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6622119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6622555Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6623136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6623592Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6624017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6624730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6625431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6626001Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6626498Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6626937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6627399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6628426Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6629745Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6631014Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6632274Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6633545Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6634808Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6636071Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6637338Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6638601Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6639857Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6640509Z dist init r=0, world=2 2022-11-23T03:25:03.6640753Z dist init r=1, world=2 2022-11-23T03:25:03.6640971Z ok (7.535s) 2022-11-23T03:25:03.6641521Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128369 2022-11-23T03:25:03.6642155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128370 2022-11-23T03:25:03.6642775Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6643218Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6643802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6644265Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6644747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6645386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6645828Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6646410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6646870Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6647309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6647967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6648654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6649219Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6649720Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6650158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6650666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6651692Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6652966Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6654227Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6655496Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6656820Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6658094Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6659351Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6660613Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6661927Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6663191Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6663782Z dist init r=0, world=2 2022-11-23T03:25:03.6664022Z dist init r=1, world=2 2022-11-23T03:25:03.6664253Z ok (7.939s) 2022-11-23T03:25:03.6664768Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128522 2022-11-23T03:25:03.6665366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128523 2022-11-23T03:25:03.6665986Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6666428Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6667012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6667471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6667909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6668546Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6668979Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6669558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6670017Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6670451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6671108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6671799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6672368Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6672864Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6673293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6673809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6674837Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6676107Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6677366Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6678681Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6679941Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6681209Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6682464Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6683718Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6684984Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6686242Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6686842Z dist init r=0, world=2 2022-11-23T03:25:03.6687082Z dist init r=1, world=2 2022-11-23T03:25:03.6687310Z ok (7.730s) 2022-11-23T03:25:03.6687849Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128675 2022-11-23T03:25:03.6688486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128676 2022-11-23T03:25:03.6689160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6689594Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6690184Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6690692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6691130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6691765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6692199Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6692780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6693310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6693739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6694405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6695093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6695663Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6696163Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6696603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6697063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6698087Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6699363Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6700632Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6701900Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6703170Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6704432Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6705755Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6707023Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6707620Z dist init r=0, world=2 2022-11-23T03:25:03.6707850Z dist init r=1, world=2 2022-11-23T03:25:03.6708078Z ok (8.132s) 2022-11-23T03:25:03.6708583Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128828 2022-11-23T03:25:03.6709191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128829 2022-11-23T03:25:03.6709877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6710315Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6710902Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6711355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6711798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6712428Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6712864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6713448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6713915Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6714352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6715015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6715702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6716274Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6716775Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6717213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6717675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6718701Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6719978Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6721237Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6722556Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6723829Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6725093Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6726408Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6727667Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6728927Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6730191Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6730906Z dist init r=0, world=2 2022-11-23T03:25:03.6731146Z dist init r=1, world=2 2022-11-23T03:25:03.6731372Z ok (7.834s) 2022-11-23T03:25:03.6731900Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128981 2022-11-23T03:25:03.6732534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128982 2022-11-23T03:25:03.6733160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6733600Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6734192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6734653Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6735089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6735722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6736151Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6736735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6737192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6737624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6738356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6739054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6739624Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6740115Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6740556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6741017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6742039Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6743363Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6744626Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6745898Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6747162Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6748424Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6749687Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6750952Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6751550Z dist init r=0, world=2 2022-11-23T03:25:03.6751789Z dist init r=1, world=2 2022-11-23T03:25:03.6752017Z ok (8.837s) 2022-11-23T03:25:03.6752520Z test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129134 2022-11-23T03:25:03.6753122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129135 2022-11-23T03:25:03.6753729Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6754230Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6754819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6755279Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6755715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6756348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6756786Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6757368Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6757819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6758260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6758968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6759664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6760233Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6760736Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6761174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6761624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6762650Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6763926Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6765189Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6766459Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6767725Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6768990Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6770308Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6771673Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6772930Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6774194Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:25:03.6774881Z dist init r=1, world=2 2022-11-23T03:25:03.6775109Z dist init r=0, world=2 2022-11-23T03:25:03.6775338Z ok (7.832s) 2022-11-23T03:25:03.6775816Z test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129287 2022-11-23T03:25:03.6776394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129288 2022-11-23T03:25:03.6777022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6777468Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6778050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6778508Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6778947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6779578Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6780015Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6780598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6781057Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6781489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6782135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6782828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6783402Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6783900Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6784344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6784807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6785152Z dist init r=0, world=2 2022-11-23T03:25:03.6785947Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6786526Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6786796Z dist init r=1, world=2 2022-11-23T03:25:03.6787667Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6788255Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6788514Z ok (7.138s) 2022-11-23T03:25:03.6788840Z test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded) 2022-11-23T03:25:03.6789280Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.004s) 2022-11-23T03:25:03.6789850Z test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129440 2022-11-23T03:25:03.6790414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129441 2022-11-23T03:25:03.6791037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6791531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6792117Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6792577Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6793015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6793637Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6794076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6794655Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6795113Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6795548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6796214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6796908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6797474Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6797961Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6798176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6798391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6798488Z dist init r=1, world=2 2022-11-23T03:25:03.6798584Z dist init r=0, world=2 2022-11-23T03:25:03.6798670Z ok (7.630s) 2022-11-23T03:25:03.6799014Z test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129593 2022-11-23T03:25:03.6799228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129594 2022-11-23T03:25:03.6799608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6799770Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6800159Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6800336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6800561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6800936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6801155Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6801550Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6801726Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6801953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6802357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6802746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6803021Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6803301Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6803518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6803780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6803882Z dist init r=1, world=2 2022-11-23T03:25:03.6803979Z dist init r=0, world=2 2022-11-23T03:25:03.6804633Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:25:03.6804728Z warnings.warn( 2022-11-23T03:25:03.6805370Z /opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead. 2022-11-23T03:25:03.6805464Z warnings.warn( 2022-11-23T03:25:03.6805546Z ok (7.234s) 2022-11-23T03:25:03.6805879Z test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129746 2022-11-23T03:25:03.6806091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129747 2022-11-23T03:25:03.6806468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6806630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6807017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6807194Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6807420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6807794Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6807956Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6808348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6808524Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6808751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6809153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6809549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6809827Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6810102Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6810315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6810735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6810830Z dist init r=1, world=2 2022-11-23T03:25:03.6810927Z dist init r=0, world=2 2022-11-23T03:25:03.6811011Z ok (7.530s) 2022-11-23T03:25:03.6811354Z test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129899 2022-11-23T03:25:03.6811564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129900 2022-11-23T03:25:03.6811950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6812111Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6812499Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6812734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6812959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6813339Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6813502Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6813887Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6814063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6814290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6814689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6815090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6815368Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6815645Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6815866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6816073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6816171Z dist init r=0, world=2 2022-11-23T03:25:03.6816266Z dist init r=1, world=2 2022-11-23T03:25:03.6816349Z ok (7.934s) 2022-11-23T03:25:03.6816688Z test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130052 2022-11-23T03:25:03.6816893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130053 2022-11-23T03:25:03.6817275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6817438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6817826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6818004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6818229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6818603Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6818766Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6819151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6819328Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6819621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6820024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6820421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6820695Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6820968Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6821184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6821390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6821488Z dist init r=0, world=2 2022-11-23T03:25:03.6821584Z dist init r=1, world=2 2022-11-23T03:25:03.6821714Z ok (7.529s) 2022-11-23T03:25:03.6822039Z test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130205 2022-11-23T03:25:03.6822249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130206 2022-11-23T03:25:03.6822630Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6822796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6823179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6823355Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6823582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6823958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6824122Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6824505Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6824680Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6824904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:03.6825301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6825699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:03.6825973Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6826247Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6826459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:03.6826671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6826768Z dist init r=0, world=2 2022-11-23T03:25:03.6826863Z dist init r=1, world=2 2022-11-23T03:25:03.6826948Z ok (7.633s) 2022-11-23T03:25:03.6827285Z test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130358 2022-11-23T03:25:03.6827663Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6827825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6828208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6828437Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6828665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6829067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:25:03.6829340Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6829553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6829648Z dist init r=0, world=1 2022-11-23T03:25:03.6829733Z ok (6.823s) 2022-11-23T03:25:03.6830073Z test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130435 2022-11-23T03:25:03.6830449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6830663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6831041Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6831222Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6831446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6831843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:25:03.6832115Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6832331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6832426Z dist init r=0, world=1 2022-11-23T03:25:03.6833088Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6833219Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6833305Z ok (6.622s) 2022-11-23T03:25:03.6833657Z test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130512 2022-11-23T03:25:03.6834032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:03.6834195Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:03.6834581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:03.6834757Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:03.6834983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:03.6835389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:25:03.6835662Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:03.6835876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:03.6835970Z dist init r=0, world=1 2022-11-23T03:25:03.6836624Z /opt/conda/lib/python3.8/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. 2022-11-23T03:25:03.6836750Z warnings.warn(message, UserWarning) 2022-11-23T03:25:03.6836826Z ok (6.623s) 2022-11-23T03:25:03.6836846Z 2022-11-23T03:25:03.6837108Z ---------------------------------------------------------------------- 2022-11-23T03:25:03.6837215Z Ran 44 tests in 330.226s 2022-11-23T03:25:03.6837221Z 2022-11-23T03:25:03.6837371Z OK (skipped=1) 2022-11-23T03:25:03.6837378Z 2022-11-23T03:25:03.6837490Z Generating XML reports... 2022-11-23T03:25:03.6838010Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221123031931.xml 2022-11-23T03:25:03.6838522Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221123031931.xml 2022-11-23T03:25:03.6838529Z 2022-11-23T03:25:03.6838973Z ##[endgroup] 2022-11-23T03:25:03.6839487Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_nkrqo773) 2022-11-23T03:25:03.6839493Z 2022-11-23T03:25:03.6839754Z Running distributed/fsdp/test_fsdp_meta ... [2022-11-23 03:25:03.603626] 2022-11-23T03:25:03.6840238Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:25:03.604332] 2022-11-23T03:25:53.0034308Z 2022-11-23T03:25:53.0035603Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta 2022-11-23T03:25:53.0037752Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_meta_0co9va8q) 2022-11-23T03:25:53.0039818Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta 2022-11-23T03:25:53.0040535Z 2022-11-23T03:25:53.0045293Z Running tests... 2022-11-23T03:25:53.0046820Z ---------------------------------------------------------------------- 2022-11-23T03:25:53.0048291Z test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130656 2022-11-23T03:25:53.0049967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130657 2022-11-23T03:25:53.0052111Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0053342Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0054988Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0056804Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0058044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:53.0059905Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0061531Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0063538Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0065050Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0066455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:53.0068319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0070276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0071859Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0073237Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0074441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:53.0075703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:53.0076636Z dist init r=1, world=2 2022-11-23T03:25:53.0077281Z dist init r=0, world=2 2022-11-23T03:25:53.0077906Z ok (5.019s) 2022-11-23T03:25:53.0079626Z test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-11-23T03:25:53.0081486Z test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130799 2022-11-23T03:25:53.0083059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130800 2022-11-23T03:25:53.0084796Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0086007Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0087613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0088868Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0090273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:53.0092444Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0093661Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0095262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0096530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0097721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:53.0099526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0101432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0103009Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0104387Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0105579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:53.0106860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:53.0107801Z dist init r=0, world=2 2022-11-23T03:25:53.0108435Z dist init r=1, world=2 2022-11-23T03:25:53.0109042Z ok (6.531s) 2022-11-23T03:25:53.0110345Z test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130952 2022-11-23T03:25:53.0111916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130953 2022-11-23T03:25:53.0113629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0114853Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0116454Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0117719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0118898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:53.0120646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0121825Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0123407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0124659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0126043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:53.0127868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0129779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0131473Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0132853Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0134074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:53.0135345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:53.0136294Z dist init r=0, world=2 2022-11-23T03:25:53.0136937Z dist init r=1, world=2 2022-11-23T03:25:53.0137733Z ok (6.532s) 2022-11-23T03:25:53.0139043Z test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 333 2022-11-23T03:25:53.0140612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 334 2022-11-23T03:25:53.0142296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0143507Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0145092Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0146340Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0147539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:53.0149270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0150477Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0152075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0153304Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0154487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:53.0156291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0158204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0159753Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0161126Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0162359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:53.0163613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:53.0164552Z dist init r=0, world=2 2022-11-23T03:25:53.0165195Z dist init r=1, world=2 2022-11-23T03:25:53.0165804Z ok (6.734s) 2022-11-23T03:25:53.0167099Z test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 486 2022-11-23T03:25:53.0168635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 487 2022-11-23T03:25:53.0170331Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0172093Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0173929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0175212Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0176416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:53.0178012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0178645Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0179338Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0179886Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0180407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:53.0181185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0182168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0182854Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0183456Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0183977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:53.0184537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:53.0184957Z dist init r=0, world=2 2022-11-23T03:25:53.0185229Z dist init r=1, world=2 2022-11-23T03:25:53.0185504Z ok (6.536s) 2022-11-23T03:25:53.0186088Z test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.002s) 2022-11-23T03:25:53.0186945Z test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-11-23T03:25:53.0187794Z test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-11-23T03:25:53.0188572Z test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-11-23T03:25:53.0189247Z test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 639 2022-11-23T03:25:53.0189795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 640 2022-11-23T03:25:53.0190410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0190843Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0191418Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0191874Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0192305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:53.0192930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0193365Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0193944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0194390Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0194883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:53.0195539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0196226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0196792Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0197287Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0197722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:53.0198183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:53.0198514Z dist init r=1, world=2 2022-11-23T03:25:53.0198749Z dist init r=0, world=2 2022-11-23T03:25:53.0198972Z ok (7.332s) 2022-11-23T03:25:53.0199479Z test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 792 2022-11-23T03:25:53.0200030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 793 2022-11-23T03:25:53.0200651Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0201078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0201658Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0202114Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0202551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:25:53.0203183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:25:53.0203624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:25:53.0204203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:25:53.0204661Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:25:53.0205086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:25:53.0205739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0206421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:25:53.0206987Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0207484Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:25:53.0207928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:25:53.0208387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:25:53.0208719Z dist init r=0, world=2 2022-11-23T03:25:53.0208954Z dist init r=1, world=2 2022-11-23T03:25:53.0209178Z ok (6.734s) 2022-11-23T03:25:53.0209637Z test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-11-23T03:25:53.0210310Z test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s) 2022-11-23T03:25:53.0210720Z 2022-11-23T03:25:53.0211001Z ---------------------------------------------------------------------- 2022-11-23T03:25:53.0211355Z Ran 14 tests in 45.431s 2022-11-23T03:25:53.0211535Z 2022-11-23T03:25:53.0211635Z OK (skipped=7) 2022-11-23T03:25:53.0211813Z 2022-11-23T03:25:53.0211941Z Generating XML reports... 2022-11-23T03:25:53.0212767Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221123032505.xml 2022-11-23T03:25:53.0213179Z 2022-11-23T03:25:53.0213677Z ##[endgroup] 2022-11-23T03:25:53.0214404Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_meta_0co9va8q) 2022-11-23T03:25:53.0214802Z 2022-11-23T03:25:53.0215125Z Running distributed/fsdp/test_fsdp_input ... [2022-11-23 03:25:53.003992] 2022-11-23T03:25:53.0215962Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:25:53.004677] 2022-11-23T03:26:09.9836983Z 2022-11-23T03:26:09.9837924Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_input 2022-11-23T03:26:09.9875868Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_input_4x94kbps) 2022-11-23T03:26:09.9878773Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input 2022-11-23T03:26:09.9879587Z 2022-11-23T03:26:09.9879872Z Running tests... 2022-11-23T03:26:09.9881113Z ---------------------------------------------------------------------- 2022-11-23T03:26:09.9882167Z test_input_type_dict (__main__.TestInput) 2022-11-23T03:26:09.9883564Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1015 2022-11-23T03:26:09.9885661Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:26:09.9886970Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:26:09.9888724Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:26:09.9890109Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:26:09.9891662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:26:09.9894446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:26:09.9896870Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:26:09.9898584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:26:09.9899765Z dist init r=0, world=1 2022-11-23T03:26:09.9904449Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:26:09.9907125Z warnings.warn( 2022-11-23T03:26:09.9907789Z ok (6.817s) 2022-11-23T03:26:09.9908553Z test_input_type_list (__main__.TestInput) 2022-11-23T03:26:09.9909909Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1092 2022-11-23T03:26:09.9912010Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:26:09.9913313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:26:09.9915038Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:26:09.9916397Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:26:09.9917739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:26:09.9920056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-11-23T03:26:09.9921787Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:26:09.9923082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:26:09.9924087Z dist init r=0, world=1 2022-11-23T03:26:09.9927710Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:26:09.9929975Z warnings.warn( 2022-11-23T03:26:09.9931409Z ok (6.227s) 2022-11-23T03:26:09.9931811Z 2022-11-23T03:26:09.9932674Z ---------------------------------------------------------------------- 2022-11-23T03:26:09.9933592Z Ran 2 tests in 13.045s 2022-11-23T03:26:09.9934027Z 2022-11-23T03:26:09.9934247Z OK 2022-11-23T03:26:09.9934592Z 2022-11-23T03:26:09.9934916Z Generating XML reports... 2022-11-23T03:26:09.9936604Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20221123032554.xml 2022-11-23T03:26:09.9937515Z 2022-11-23T03:26:09.9938342Z ##[endgroup] 2022-11-23T03:26:09.9940115Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_input_4x94kbps) 2022-11-23T03:26:09.9940996Z 2022-11-23T03:26:09.9941338Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-11-23 03:26:09.984034] 2022-11-23T03:26:09.9942101Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:26:09.984716] 2022-11-23T03:27:28.7507542Z 2022-11-23T03:27:28.7508813Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_grad_acc 2022-11-23T03:27:28.7511113Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_30_csn64) 2022-11-23T03:27:28.7513221Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc 2022-11-23T03:27:28.7513944Z 2022-11-23T03:27:28.7514206Z Running tests... 2022-11-23T03:27:28.7515309Z ---------------------------------------------------------------------- 2022-11-23T03:27:28.7516957Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc) 2022-11-23T03:27:28.7519141Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1236 2022-11-23T03:27:28.7520901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1237 2022-11-23T03:27:28.7522990Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7524306Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7526028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7527351Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7528621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7531071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7532633Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7535564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7537088Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7538494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7540648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7542611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7544131Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7545459Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7546617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7548091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7548980Z dist init r=0, world=2 2022-11-23T03:27:28.7549595Z dist init r=1, world=2 2022-11-23T03:27:28.7552867Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7554924Z warnings.warn( 2022-11-23T03:27:28.7558143Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7560199Z warnings.warn( 2022-11-23T03:27:28.7560781Z ok (9.550s) 2022-11-23T03:27:28.7562155Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc) 2022-11-23T03:27:28.7563908Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1389 2022-11-23T03:27:28.7565189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1390 2022-11-23T03:27:28.7566829Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7567975Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7569533Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7570949Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7572103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7573791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7574931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7576480Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7577690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7578836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7580762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7582609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7584112Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7585433Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7586597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7587801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7588705Z dist init r=0, world=2 2022-11-23T03:27:28.7591937Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7594131Z warnings.warn( 2022-11-23T03:27:28.7594739Z dist init r=1, world=2 2022-11-23T03:27:28.7597952Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7599970Z warnings.warn( 2022-11-23T03:27:28.7600547Z ok (7.737s) 2022-11-23T03:27:28.7601946Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc) 2022-11-23T03:27:28.7603711Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1542 2022-11-23T03:27:28.7604980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1543 2022-11-23T03:27:28.7606602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7608069Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7609619Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7610955Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7612084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7613780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7614931Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7616463Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7617659Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7618799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7620547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7622362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7623860Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7625348Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7626520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7627726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7628626Z dist init r=1, world=2 2022-11-23T03:27:28.7629238Z dist init r=0, world=2 2022-11-23T03:27:28.7632465Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7634486Z warnings.warn( 2022-11-23T03:27:28.7637824Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7639862Z warnings.warn( 2022-11-23T03:27:28.7640435Z ok (7.636s) 2022-11-23T03:27:28.7641805Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc) 2022-11-23T03:27:28.7643550Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1695 2022-11-23T03:27:28.7644818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1696 2022-11-23T03:27:28.7646466Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7647610Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7649133Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7650335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7652067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7653768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7654905Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7656434Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7657647Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7658783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7660497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7662326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7663829Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7665137Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7665907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7666440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7666783Z dist init r=1, world=2 2022-11-23T03:27:28.7667085Z dist init r=0, world=2 2022-11-23T03:27:28.7667310Z ok (4.430s) 2022-11-23T03:27:28.7667825Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc) 2022-11-23T03:27:28.7668481Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1838 2022-11-23T03:27:28.7668962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1839 2022-11-23T03:27:28.7669581Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7670016Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7670591Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7671103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7671538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7672170Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7672605Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7673180Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7673627Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7674047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7674698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7675389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7675951Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7676444Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7676881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7677336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7677678Z dist init r=1, world=2 2022-11-23T03:27:28.7677901Z dist init r=0, world=2 2022-11-23T03:27:28.7678124Z ok (4.433s) 2022-11-23T03:27:28.7678643Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc) 2022-11-23T03:27:28.7679303Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1981 2022-11-23T03:27:28.7679790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1982 2022-11-23T03:27:28.7680401Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7680838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7681409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7681865Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7682295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7682924Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7683355Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7683992Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7684448Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7684871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7685527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7686207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7686770Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7687266Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7687697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7688208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7688546Z dist init r=1, world=2 2022-11-23T03:27:28.7688773Z dist init r=0, world=2 2022-11-23T03:27:28.7688999Z ok (4.433s) 2022-11-23T03:27:28.7689517Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc) 2022-11-23T03:27:28.7690176Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2124 2022-11-23T03:27:28.7690759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2125 2022-11-23T03:27:28.7691420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7691938Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7692642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7693187Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7693703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7694458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7694972Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7695670Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7696213Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7696725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7697522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7698346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7699029Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7699631Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7700156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7700705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7701096Z dist init r=0, world=2 2022-11-23T03:27:28.7701377Z dist init r=1, world=2 2022-11-23T03:27:28.7702917Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7703844Z warnings.warn( 2022-11-23T03:27:28.7705289Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7706154Z warnings.warn( 2022-11-23T03:27:28.7706374Z ok (7.634s) 2022-11-23T03:27:28.7706895Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc) 2022-11-23T03:27:28.7707604Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2277 2022-11-23T03:27:28.7708083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2278 2022-11-23T03:27:28.7708695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7709121Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7709697Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7710156Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7710585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7711220Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7711652Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7712229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7712676Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7713105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7713752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7714437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7714999Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7715499Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7715936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7716386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7716723Z dist init r=0, world=2 2022-11-23T03:27:28.7716959Z dist init r=1, world=2 2022-11-23T03:27:28.7718163Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7718920Z warnings.warn( 2022-11-23T03:27:28.7720182Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7720946Z warnings.warn( 2022-11-23T03:27:28.7721167Z ok (7.737s) 2022-11-23T03:27:28.7721690Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc) 2022-11-23T03:27:28.7722350Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2430 2022-11-23T03:27:28.7722819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2431 2022-11-23T03:27:28.7723486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7723917Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7724498Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7724952Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7725388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7726017Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7726449Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7727018Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7727480Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7727912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7728562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7729248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7729812Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7730305Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7730802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7731261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7731679Z dist init r=1, world=2 2022-11-23T03:27:28.7733114Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7734037Z warnings.warn( 2022-11-23T03:27:28.7734315Z dist init r=0, world=2 2022-11-23T03:27:28.7735872Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:27:28.7736801Z warnings.warn( 2022-11-23T03:27:28.7737064Z ok (7.636s) 2022-11-23T03:27:28.7737692Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc) 2022-11-23T03:27:28.7738477Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2583 2022-11-23T03:27:28.7739065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2584 2022-11-23T03:27:28.7739807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7740327Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7741030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7741646Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7742167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7742936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7743445Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7744141Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7744686Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7745204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7745977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7746668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7747229Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7747715Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7748148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7748605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7748943Z dist init r=1, world=2 2022-11-23T03:27:28.7749179Z dist init r=0, world=2 2022-11-23T03:27:28.7749403Z ok (4.434s) 2022-11-23T03:27:28.7749918Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc) 2022-11-23T03:27:28.7750569Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2726 2022-11-23T03:27:28.7751051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2727 2022-11-23T03:27:28.7751664Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7752099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7752679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7753134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7753573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7754192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7754630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7755270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7755730Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7756160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7756819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7757502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7758065Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7758553Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7758987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7759498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7759838Z dist init r=1, world=2 2022-11-23T03:27:28.7760073Z dist init r=0, world=2 2022-11-23T03:27:28.7760300Z ok (4.432s) 2022-11-23T03:27:28.7760811Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc) 2022-11-23T03:27:28.7761470Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2869 2022-11-23T03:27:28.7761949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2870 2022-11-23T03:27:28.7762564Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7762997Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7763586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7764039Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7764469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:27:28.7765084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:27:28.7765519Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:27:28.7766095Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:27:28.7766544Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:27:28.7766976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:27:28.7767633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7768321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:27:28.7768884Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7769370Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:27:28.7769804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:27:28.7770265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:27:28.7770863Z dist init r=1, world=2 2022-11-23T03:27:28.7771110Z dist init r=0, world=2 2022-11-23T03:27:28.7771360Z ok (4.631s) 2022-11-23T03:27:28.7771516Z 2022-11-23T03:27:28.7771851Z ---------------------------------------------------------------------- 2022-11-23T03:27:28.7772232Z Ran 12 tests in 74.727s 2022-11-23T03:27:28.7772500Z 2022-11-23T03:27:28.7772598Z OK 2022-11-23T03:27:28.7772742Z 2022-11-23T03:27:28.7772877Z Generating XML reports... 2022-11-23T03:27:28.7773576Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221123032611.xml 2022-11-23T03:27:28.7773960Z 2022-11-23T03:27:28.7774340Z ##[endgroup] 2022-11-23T03:27:28.7775069Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_30_csn64) 2022-11-23T03:27:28.7775486Z 2022-11-23T03:27:28.7775841Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-11-23 03:27:28.751852] 2022-11-23T03:27:28.7776718Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:27:28.752525] 2022-11-23T03:28:31.3531296Z 2022-11-23T03:28:31.3532716Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_freezing_weights 2022-11-23T03:28:31.3537307Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_a6ha3alp) 2022-11-23T03:28:31.3539743Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights 2022-11-23T03:28:31.3540752Z 2022-11-23T03:28:31.3541027Z Running tests... 2022-11-23T03:28:31.3542174Z ---------------------------------------------------------------------- 2022-11-23T03:28:31.3543938Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3079 2022-11-23T03:28:31.3546131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3080 2022-11-23T03:28:31.3548673Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3550204Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3552755Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3554357Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3555638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:28:31.3557848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3559252Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3561354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3562860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3564211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:28:31.3566199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3568220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3569895Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3571502Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3573225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:28:31.3574553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:28:31.3575912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3577849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3578866Z dist init r=1, world=2 2022-11-23T03:28:31.3579571Z dist init r=0, world=2 2022-11-23T03:28:31.3580234Z ok (8.534s) 2022-11-23T03:28:31.3581789Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3232 2022-11-23T03:28:31.3583617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3233 2022-11-23T03:28:31.3585435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3586689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3588376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3589919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3591176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:28:31.3592997Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3594239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3595928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3597228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3598456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:28:31.3600335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3602221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3603753Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3605086Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3606263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:28:31.3607505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:28:31.3608745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3609983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3611066Z dist init r=0, world=2 2022-11-23T03:28:31.3614478Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:28:31.3616598Z warnings.warn( 2022-11-23T03:28:31.3617223Z dist init r=1, world=2 2022-11-23T03:28:31.3620593Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:28:31.3622641Z warnings.warn( 2022-11-23T03:28:31.3623229Z ok (6.636s) 2022-11-23T03:28:31.3624882Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3385 2022-11-23T03:28:31.3626614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3386 2022-11-23T03:28:31.3628264Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3629434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3631002Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3632226Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3633382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:28:31.3635088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3636397Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3637951Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3639171Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3640319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:28:31.3642088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3643963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3645488Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3646826Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3648014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:28:31.3649236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:28:31.3650477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3652670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3653605Z dist init r=0, world=2 2022-11-23T03:28:31.3654234Z dist init r=1, world=2 2022-11-23T03:28:31.3654844Z ok (6.648s) 2022-11-23T03:28:31.3656299Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3538 2022-11-23T03:28:31.3658025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3539 2022-11-23T03:28:31.3659746Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3660911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3662482Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3663696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3664543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:28:31.3665262Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3665686Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3666261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3666796Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3667230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:28:31.3667885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3668570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3669132Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3669619Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3670055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:28:31.3670514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:28:31.3670977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3671561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3671906Z dist init r=1, world=2 2022-11-23T03:28:31.3673126Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:28:31.3673900Z warnings.warn( 2022-11-23T03:28:31.3674132Z dist init r=0, world=2 2022-11-23T03:28:31.3675329Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:28:31.3676093Z warnings.warn( 2022-11-23T03:28:31.3676315Z ok (7.742s) 2022-11-23T03:28:31.3676857Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3691 2022-11-23T03:28:31.3677490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3692 2022-11-23T03:28:31.3678105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3678541Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3679124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3679574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3680010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:28:31.3680634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3681063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3681643Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3682095Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3682527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:28:31.3683240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3683925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3684490Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3684986Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3685419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:28:31.3685874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:28:31.3686337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3686801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3687138Z dist init r=1, world=2 2022-11-23T03:28:31.3687375Z dist init r=0, world=2 2022-11-23T03:28:31.3687656Z ok (8.439s) 2022-11-23T03:28:31.3688203Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3964 2022-11-23T03:28:31.3688839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3965 2022-11-23T03:28:31.3689461Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3689899Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3690474Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3691004Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3691445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:28:31.3692151Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3692672Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3693375Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3693922Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3694435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:28:31.3695223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3696049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3696727Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3697332Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3697874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:28:31.3698429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:28:31.3698988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3699542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3699952Z dist init r=0, world=2 2022-11-23T03:28:31.3701481Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:28:31.3702424Z warnings.warn( 2022-11-23T03:28:31.3702702Z dist init r=1, world=2 2022-11-23T03:28:31.3704160Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:28:31.3705027Z warnings.warn( 2022-11-23T03:28:31.3705248Z ok (6.931s) 2022-11-23T03:28:31.3705792Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4179 2022-11-23T03:28:31.3706489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4180 2022-11-23T03:28:31.3707098Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3707533Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3708110Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3708568Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3709002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:28:31.3709629Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3710063Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3710634Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3711092Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3711520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:28:31.3712169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3712851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3713416Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3713913Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3714353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:28:31.3714803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:28:31.3715269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3715738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3716079Z dist init r=0, world=2 2022-11-23T03:28:31.3716311Z dist init r=1, world=2 2022-11-23T03:28:31.3716535Z ok (6.937s) 2022-11-23T03:28:31.3717069Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4332 2022-11-23T03:28:31.3717706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4333 2022-11-23T03:28:31.3718318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3718749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3719389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3719849Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3720280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:28:31.3720917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:28:31.3721344Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:28:31.3721919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:28:31.3722371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:28:31.3722801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:28:31.3723507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3724192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:28:31.3724760Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3725249Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:28:31.3725688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:28:31.3726144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:28:31.3726605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3727071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:28:31.3727416Z dist init r=0, world=2 2022-11-23T03:28:31.3727658Z dist init r=1, world=2 2022-11-23T03:28:31.3728857Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:28:31.3729628Z warnings.warn( 2022-11-23T03:28:31.3730893Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:28:31.3731664Z warnings.warn( 2022-11-23T03:28:31.3731898Z ok (6.735s) 2022-11-23T03:28:31.3732061Z 2022-11-23T03:28:31.3732391Z ---------------------------------------------------------------------- 2022-11-23T03:28:31.3732757Z Ran 8 tests in 58.604s 2022-11-23T03:28:31.3732931Z 2022-11-23T03:28:31.3733020Z OK 2022-11-23T03:28:31.3733157Z 2022-11-23T03:28:31.3733284Z Generating XML reports... 2022-11-23T03:28:31.3734018Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20221123032730.xml 2022-11-23T03:28:31.3734441Z 2022-11-23T03:28:31.3734820Z ##[endgroup] 2022-11-23T03:28:31.3735602Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_a6ha3alp) 2022-11-23T03:28:31.3736048Z 2022-11-23T03:28:31.3736378Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-11-23 03:28:31.353987] 2022-11-23T03:28:31.3737316Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:28:31.354663] 2022-11-23T03:29:29.4708824Z 2022-11-23T03:29:29.4709670Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_exec_order 2022-11-23T03:29:29.4711825Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_exec_order_vtpohc9h) 2022-11-23T03:29:29.4715265Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order 2022-11-23T03:29:29.4716045Z 2022-11-23T03:29:29.4716277Z Running tests... 2022-11-23T03:29:29.4717397Z ---------------------------------------------------------------------- 2022-11-23T03:29:29.4718635Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) 2022-11-23T03:29:29.4720672Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4614 2022-11-23T03:29:29.4722563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4615 2022-11-23T03:29:29.4724244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4725433Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4727021Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4728240Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4729433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:29:29.4731621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4733835Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4735198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4736769Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4737997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4739149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:29:29.4740893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4742393Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4743712Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4744865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:29:29.4746092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:29:29.4746995Z dist init r=1, world=2 2022-11-23T03:29:29.4750244Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4752392Z warnings.warn( 2022-11-23T03:29:29.4753006Z dist init r=0, world=2 2022-11-23T03:29:29.4756542Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4758607Z warnings.warn( 2022-11-23T03:29:29.4759194Z ok (7.022s) 2022-11-23T03:29:29.4760190Z test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) 2022-11-23T03:29:29.4762155Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4759 2022-11-23T03:29:29.4763558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4760 2022-11-23T03:29:29.4765174Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4766332Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4768059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4769269Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4770429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:29:29.4772294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4773728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4775438Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4776654Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4777809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:29:29.4779573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4781406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4782914Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4784234Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4785398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:29:29.4786603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:29:29.4787491Z dist init r=0, world=2 2022-11-23T03:29:29.4790710Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4792757Z warnings.warn( 2022-11-23T03:29:29.4793364Z dist init r=1, world=2 2022-11-23T03:29:29.4796566Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4798587Z warnings.warn( 2022-11-23T03:29:29.4799171Z ok (6.635s) 2022-11-23T03:29:29.4800436Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2022-11-23T03:29:29.4802472Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4904 2022-11-23T03:29:29.4803892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4905 2022-11-23T03:29:29.4805526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4806689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4808238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4809439Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4810672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:29:29.4812366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4813662Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4815215Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4816410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4817551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:29:29.4819288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4821129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4822628Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4823926Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4825102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:29:29.4826322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:29:29.4827229Z dist init r=0, world=2 2022-11-23T03:29:29.4830442Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4832458Z warnings.warn( 2022-11-23T03:29:29.4833075Z dist init r=1, world=2 2022-11-23T03:29:29.4836282Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4838292Z warnings.warn( 2022-11-23T03:29:29.4838875Z ok (6.635s) 2022-11-23T03:29:29.4839929Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2022-11-23T03:29:29.4841958Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5057 2022-11-23T03:29:29.4843362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5058 2022-11-23T03:29:29.4844995Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4846305Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4847864Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4849070Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4850206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:29:29.4852213Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4853364Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4854906Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4856103Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4857447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:29:29.4859179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4860085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4860752Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4861352Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4861880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:29:29.4862432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:29:29.4862840Z dist init r=1, world=2 2022-11-23T03:29:29.4863131Z dist init r=0, world=2 2022-11-23T03:29:29.4864594Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4865525Z warnings.warn( 2022-11-23T03:29:29.4866970Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4867818Z warnings.warn( 2022-11-23T03:29:29.4868037Z ok (7.337s) 2022-11-23T03:29:29.4868451Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder) 2022-11-23T03:29:29.4869217Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5210 2022-11-23T03:29:29.4869746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5211 2022-11-23T03:29:29.4870362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4870796Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4871382Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4871826Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4872319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:29:29.4872958Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4873390Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4873968Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4874423Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4874857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:29:29.4875503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4876188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4876806Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4877305Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4877743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:29:29.4878203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:29:29.4878545Z dist init r=0, world=2 2022-11-23T03:29:29.4879753Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4880513Z warnings.warn( 2022-11-23T03:29:29.4880745Z dist init r=1, world=2 2022-11-23T03:29:29.4881937Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4882685Z warnings.warn( 2022-11-23T03:29:29.4882910Z ok (6.637s) 2022-11-23T03:29:29.4883320Z test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder) 2022-11-23T03:29:29.4884080Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5363 2022-11-23T03:29:29.4884608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5364 2022-11-23T03:29:29.4885229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4885664Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4886237Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4886690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4887123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:29:29.4887751Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4888187Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4888764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4889278Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4889707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:29:29.4890366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4891131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4891797Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4892396Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4892920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:29:29.4893474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:29:29.4893953Z dist init r=0, world=2 2022-11-23T03:29:29.4895408Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4896337Z warnings.warn( 2022-11-23T03:29:29.4896618Z dist init r=1, world=2 2022-11-23T03:29:29.4898063Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4898999Z warnings.warn( 2022-11-23T03:29:29.4899264Z ok (6.540s) 2022-11-23T03:29:29.4899831Z test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5516 2022-11-23T03:29:29.4900511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5517 2022-11-23T03:29:29.4901255Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4901772Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4902468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4903021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4903549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:29:29.4904314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4904837Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4905532Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4906072Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4906591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:29:29.4907383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4908088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4908713Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4909218Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4909658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:29:29.4910116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:29:29.4910449Z dist init r=0, world=2 2022-11-23T03:29:29.4911647Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4912408Z warnings.warn( 2022-11-23T03:29:29.4912688Z dist init r=1, world=2 2022-11-23T03:29:29.4913890Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4914654Z warnings.warn( 2022-11-23T03:29:29.4914876Z ok (6.638s) 2022-11-23T03:29:29.4915353Z test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5669 2022-11-23T03:29:29.4915921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5670 2022-11-23T03:29:29.4916536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4916971Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4917555Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4918012Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4918452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:29:29.4919079Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:29:29.4919511Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:29:29.4920094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:29:29.4920539Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:29:29.4920976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:29:29.4921632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4922318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:29:29.4922890Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4923385Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:29:29.4923823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:29:29.4924276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:29:29.4924619Z dist init r=0, world=2 2022-11-23T03:29:29.4925879Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4926643Z warnings.warn( 2022-11-23T03:29:29.4926876Z dist init r=1, world=2 2022-11-23T03:29:29.4928079Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:29:29.4928828Z warnings.warn( 2022-11-23T03:29:29.4929113Z ok (6.636s) 2022-11-23T03:29:29.4929251Z 2022-11-23T03:29:29.4929531Z ---------------------------------------------------------------------- 2022-11-23T03:29:29.4929849Z Ran 8 tests in 54.083s 2022-11-23T03:29:29.4930015Z 2022-11-23T03:29:29.4930084Z OK 2022-11-23T03:29:29.4930201Z 2022-11-23T03:29:29.4930311Z Generating XML reports... 2022-11-23T03:29:29.4930987Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221123032833.xml 2022-11-23T03:29:29.4931353Z 2022-11-23T03:29:29.4931888Z ##[endgroup] 2022-11-23T03:29:29.4932626Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_exec_order_vtpohc9h) 2022-11-23T03:29:29.4933043Z 2022-11-23T03:29:29.4933374Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2022-11-23 03:29:29.471818] 2022-11-23T03:29:29.4934223Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:29:29.472507] 2022-11-23T03:31:40.6913933Z 2022-11-23T03:31:40.6914999Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm_hooks 2022-11-23T03:31:40.6917300Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_wxc1jy08) 2022-11-23T03:31:40.6922070Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks 2022-11-23T03:31:40.6922891Z 2022-11-23T03:31:40.6923157Z Running tests... 2022-11-23T03:31:40.6924275Z ---------------------------------------------------------------------- 2022-11-23T03:31:40.6925745Z test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:31:40.6927538Z test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:31:40.6929359Z test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:31:40.6931295Z test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:31:40.6933074Z test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:31:40.6934863Z test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: BFloat16 is only supported by CUDA 11+ (0.001s) 2022-11-23T03:31:40.6936516Z test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.6939051Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5889 2022-11-23T03:31:40.6940507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5890 2022-11-23T03:31:40.6942343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.6943570Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.6945160Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.6946413Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.6947610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.6949335Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.6950802Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.6952410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.6953623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.6954802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.6956574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.6958464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.6959989Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.6961332Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.6962529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.6963770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.6964699Z dist init r=0, world=2 2022-11-23T03:31:40.6965346Z dist init r=1, world=2 2022-11-23T03:31:40.6965948Z ok (7.139s) 2022-11-23T03:31:40.6967032Z test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.6969143Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6042 2022-11-23T03:31:40.6970759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6043 2022-11-23T03:31:40.6972500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.6973753Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.6975396Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.6976690Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.6977924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.6979707Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.6980928Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.6982547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.6983836Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.6985062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.6987096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.6989067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.6990668Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.6992070Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.6993303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.6994588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.6995557Z dist init r=1, world=2 2022-11-23T03:31:40.6996217Z dist init r=0, world=2 2022-11-23T03:31:40.6996840Z ok (6.736s) 2022-11-23T03:31:40.6997979Z test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7000269Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6195 2022-11-23T03:31:40.7001769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6196 2022-11-23T03:31:40.7003485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7004714Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7006353Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7007637Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7008854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7011898Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7013154Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7014777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7016068Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7017283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7018623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7019459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7020143Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7020746Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7021278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7021827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7022242Z dist init r=1, world=2 2022-11-23T03:31:40.7022533Z dist init r=0, world=2 2022-11-23T03:31:40.7022807Z ok (6.530s) 2022-11-23T03:31:40.7023335Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7024243Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6348 2022-11-23T03:31:40.7024848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6349 2022-11-23T03:31:40.7025590Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7026205Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7026918Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7027479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7028008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7028698Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7029144Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7029721Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7030185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7030627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7031346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7032034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7032604Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7033106Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7033542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7034008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7034349Z dist init r=1, world=2 2022-11-23T03:31:40.7034622Z Communication hook should not be `None` 2022-11-23T03:31:40.7034946Z Communication hook state should not be `None` 2022-11-23T03:31:40.7035228Z dist init r=0, world=2 2022-11-23T03:31:40.7035496Z Communication hook should not be `None` 2022-11-23T03:31:40.7035812Z Communication hook state should not be `None` 2022-11-23T03:31:40.7036349Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7036727Z return func(*args, **kwargs) 2022-11-23T03:31:40.7037278Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7037661Z _check_comm_hook( 2022-11-23T03:31:40.7038184Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7038642Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7039208Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7039584Z traceback.print_stack() 2022-11-23T03:31:40.7040098Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7040471Z return func(*args, **kwargs) 2022-11-23T03:31:40.7041015Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7041383Z _check_comm_hook( 2022-11-23T03:31:40.7041906Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7042272Z p_assert( 2022-11-23T03:31:40.7042756Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7043129Z traceback.print_stack() 2022-11-23T03:31:40.7043643Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7044019Z return func(*args, **kwargs) 2022-11-23T03:31:40.7044621Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7045000Z _check_comm_hook( 2022-11-23T03:31:40.7045522Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7045988Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7046557Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7046925Z traceback.print_stack() 2022-11-23T03:31:40.7047436Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7047803Z return func(*args, **kwargs) 2022-11-23T03:31:40.7048349Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7048723Z _check_comm_hook( 2022-11-23T03:31:40.7049308Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7049672Z p_assert( 2022-11-23T03:31:40.7050152Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7050515Z traceback.print_stack() 2022-11-23T03:31:40.7050812Z ok (6.536s) 2022-11-23T03:31:40.7051270Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7052166Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6501 2022-11-23T03:31:40.7052785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6502 2022-11-23T03:31:40.7053528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7054068Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7054765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7055327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7055855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7056624Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7057152Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7057848Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7058400Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7058933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7059724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7060555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7061241Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7061842Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7062371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7062934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7063350Z dist init r=0, world=2 2022-11-23T03:31:40.7063668Z Communication hook should not be `None` 2022-11-23T03:31:40.7064044Z Communication hook state should not be `None` 2022-11-23T03:31:40.7064774Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7065228Z return func(*args, **kwargs) 2022-11-23T03:31:40.7065890Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7066346Z _check_comm_hook( 2022-11-23T03:31:40.7066963Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7067521Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7068210Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7068611Z traceback.print_stack() 2022-11-23T03:31:40.7069123Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7069497Z return func(*args, **kwargs) 2022-11-23T03:31:40.7070102Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7070473Z _check_comm_hook( 2022-11-23T03:31:40.7070994Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7071356Z p_assert( 2022-11-23T03:31:40.7071836Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7072206Z traceback.print_stack() 2022-11-23T03:31:40.7072460Z dist init r=1, world=2 2022-11-23T03:31:40.7072723Z Communication hook should not be `None` 2022-11-23T03:31:40.7073042Z Communication hook state should not be `None` 2022-11-23T03:31:40.7073580Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7073949Z return func(*args, **kwargs) 2022-11-23T03:31:40.7074494Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7074874Z _check_comm_hook( 2022-11-23T03:31:40.7075384Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7075846Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7076412Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7076780Z traceback.print_stack() 2022-11-23T03:31:40.7077288Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7077659Z return func(*args, **kwargs) 2022-11-23T03:31:40.7078200Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7078565Z _check_comm_hook( 2022-11-23T03:31:40.7079094Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7079456Z p_assert( 2022-11-23T03:31:40.7079934Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7080304Z traceback.print_stack() 2022-11-23T03:31:40.7080548Z ok (6.535s) 2022-11-23T03:31:40.7080985Z test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7081747Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6654 2022-11-23T03:31:40.7082261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6655 2022-11-23T03:31:40.7082877Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7083382Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7083977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7084438Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7084875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7085500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7085943Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7086528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7086986Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7087422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7088143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7088838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7089412Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7089905Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7090349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7090878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7091238Z dist init r=1, world=2 2022-11-23T03:31:40.7091570Z Communication hook should not be `None` 2022-11-23T03:31:40.7091944Z Communication hook state should not be `None` 2022-11-23T03:31:40.7092267Z dist init r=0, world=2 2022-11-23T03:31:40.7092600Z Communication hook should not be `None` 2022-11-23T03:31:40.7092978Z Communication hook state should not be `None` 2022-11-23T03:31:40.7093629Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7094077Z return func(*args, **kwargs) 2022-11-23T03:31:40.7094733Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7095179Z _check_comm_hook( 2022-11-23T03:31:40.7095805Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7096362Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7097047Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7097496Z traceback.print_stack() 2022-11-23T03:31:40.7098115Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7098562Z return func(*args, **kwargs) 2022-11-23T03:31:40.7099212Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7099664Z _check_comm_hook( 2022-11-23T03:31:40.7100294Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7100727Z p_assert( 2022-11-23T03:31:40.7101299Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7101741Z traceback.print_stack() 2022-11-23T03:31:40.7102353Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7102786Z return func(*args, **kwargs) 2022-11-23T03:31:40.7103517Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7103980Z _check_comm_hook( 2022-11-23T03:31:40.7104610Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7105177Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7105858Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7106290Z traceback.print_stack() 2022-11-23T03:31:40.7106903Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7107348Z return func(*args, **kwargs) 2022-11-23T03:31:40.7108003Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7108463Z _check_comm_hook( 2022-11-23T03:31:40.7109006Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7109426Z p_assert( 2022-11-23T03:31:40.7109900Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7110267Z traceback.print_stack() 2022-11-23T03:31:40.7110514Z ok (6.534s) 2022-11-23T03:31:40.7110954Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7111709Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6807 2022-11-23T03:31:40.7112224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6808 2022-11-23T03:31:40.7112840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7113273Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7113869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7114334Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7114774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7115409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7115846Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7116432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7116881Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7117323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7117990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7118682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7119251Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7119755Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7120199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7120663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7121001Z dist init r=1, world=2 2022-11-23T03:31:40.7121277Z Communication hook should not be `None` 2022-11-23T03:31:40.7121593Z Communication hook state should not be `None` 2022-11-23T03:31:40.7122214Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7122595Z return func(*args, **kwargs) 2022-11-23T03:31:40.7123145Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7123517Z _check_comm_hook( 2022-11-23T03:31:40.7124042Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7124506Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7125074Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7125448Z traceback.print_stack() 2022-11-23T03:31:40.7125957Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7126326Z return func(*args, **kwargs) 2022-11-23T03:31:40.7126865Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7127299Z _check_comm_hook( 2022-11-23T03:31:40.7127823Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7128185Z p_assert( 2022-11-23T03:31:40.7128661Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7129031Z traceback.print_stack() 2022-11-23T03:31:40.7129279Z dist init r=0, world=2 2022-11-23T03:31:40.7129551Z Communication hook should not be `None` 2022-11-23T03:31:40.7129866Z Communication hook state should not be `None` 2022-11-23T03:31:40.7130404Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7130897Z return func(*args, **kwargs) 2022-11-23T03:31:40.7131446Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7131827Z _check_comm_hook( 2022-11-23T03:31:40.7132347Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7132812Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7133378Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7133747Z traceback.print_stack() 2022-11-23T03:31:40.7134254Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7134625Z return func(*args, **kwargs) 2022-11-23T03:31:40.7135155Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7135531Z _check_comm_hook( 2022-11-23T03:31:40.7136053Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7136416Z p_assert( 2022-11-23T03:31:40.7136890Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7137259Z traceback.print_stack() 2022-11-23T03:31:40.7137497Z ok (6.539s) 2022-11-23T03:31:40.7137933Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7138689Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6960 2022-11-23T03:31:40.7139199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6961 2022-11-23T03:31:40.7139817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7140258Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7140925Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7141389Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7141821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7142460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7142897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7143485Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7143942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7144378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7145105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7145791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7146363Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7146863Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7147305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7147770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7148116Z dist init r=0, world=2 2022-11-23T03:31:40.7148394Z Communication hook should not be `None` 2022-11-23T03:31:40.7148706Z Communication hook state should not be `None` 2022-11-23T03:31:40.7149238Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7149615Z return func(*args, **kwargs) 2022-11-23T03:31:40.7150158Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7150536Z _check_comm_hook( 2022-11-23T03:31:40.7151058Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7151523Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7152081Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7152451Z traceback.print_stack() 2022-11-23T03:31:40.7152960Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7153333Z return func(*args, **kwargs) 2022-11-23T03:31:40.7153876Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7154260Z _check_comm_hook( 2022-11-23T03:31:40.7154772Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7155137Z p_assert( 2022-11-23T03:31:40.7155617Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7155984Z traceback.print_stack() 2022-11-23T03:31:40.7156246Z dist init r=1, world=2 2022-11-23T03:31:40.7156522Z Communication hook should not be `None` 2022-11-23T03:31:40.7156833Z Communication hook state should not be `None` 2022-11-23T03:31:40.7157370Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7157742Z return func(*args, **kwargs) 2022-11-23T03:31:40.7158289Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7158734Z _check_comm_hook( 2022-11-23T03:31:40.7159261Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7159724Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7160287Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7160655Z traceback.print_stack() 2022-11-23T03:31:40.7161168Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7161539Z return func(*args, **kwargs) 2022-11-23T03:31:40.7162084Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7162460Z _check_comm_hook( 2022-11-23T03:31:40.7162979Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7163387Z p_assert( 2022-11-23T03:31:40.7163867Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7164234Z traceback.print_stack() 2022-11-23T03:31:40.7164479Z ok (6.637s) 2022-11-23T03:31:40.7164919Z test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7165686Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7113 2022-11-23T03:31:40.7166195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7114 2022-11-23T03:31:40.7166815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7167257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7167857Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7168320Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7168759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7169395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7169831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7170407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7170919Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7171362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7172032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7172723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7173293Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7173797Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7174230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7174696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7175037Z dist init r=1, world=2 2022-11-23T03:31:40.7175307Z Communication hook should not be `None` 2022-11-23T03:31:40.7175629Z Communication hook state should not be `None` 2022-11-23T03:31:40.7176167Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7176617Z return func(*args, **kwargs) 2022-11-23T03:31:40.7177159Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7177541Z _check_comm_hook( 2022-11-23T03:31:40.7178060Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7178524Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7179091Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7179462Z traceback.print_stack() 2022-11-23T03:31:40.7179962Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7180335Z return func(*args, **kwargs) 2022-11-23T03:31:40.7180877Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7181316Z _check_comm_hook( 2022-11-23T03:31:40.7181839Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7182206Z p_assert( 2022-11-23T03:31:40.7182684Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7183043Z traceback.print_stack() 2022-11-23T03:31:40.7183298Z dist init r=0, world=2 2022-11-23T03:31:40.7183570Z Communication hook should not be `None` 2022-11-23T03:31:40.7183891Z Communication hook state should not be `None` 2022-11-23T03:31:40.7184428Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7184801Z return func(*args, **kwargs) 2022-11-23T03:31:40.7185337Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7185726Z _check_comm_hook( 2022-11-23T03:31:40.7186247Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 652, in _check_comm_hook 2022-11-23T03:31:40.7186711Z p_assert(comm_hook is not None, "Communication hook should not be `None`") 2022-11-23T03:31:40.7187279Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7187651Z traceback.print_stack() 2022-11-23T03:31:40.7188162Z File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context 2022-11-23T03:31:40.7188526Z return func(*args, **kwargs) 2022-11-23T03:31:40.7189070Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 515, in _post_backward_hook 2022-11-23T03:31:40.7189447Z _check_comm_hook( 2022-11-23T03:31:40.7189972Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 653, in _check_comm_hook 2022-11-23T03:31:40.7190341Z p_assert( 2022-11-23T03:31:40.7190819Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert 2022-11-23T03:31:40.7191192Z traceback.print_stack() 2022-11-23T03:31:40.7191428Z ok (6.536s) 2022-11-23T03:31:40.7191934Z test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7266 2022-11-23T03:31:40.7192533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7267 2022-11-23T03:31:40.7193153Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7193596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7194186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7194710Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7195148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7195789Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7196232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7196819Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7197281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7197724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7198387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7199141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7199702Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7200210Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7200656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7201124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7201474Z dist init r=1, world=2 2022-11-23T03:31:40.7201719Z dist init r=0, world=2 2022-11-23T03:31:40.7201939Z ok (6.733s) 2022-11-23T03:31:40.7202442Z test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7419 2022-11-23T03:31:40.7203034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7420 2022-11-23T03:31:40.7203667Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7204113Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7204701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7205162Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7205595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7206233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7206677Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7207265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7207729Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7208170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7208831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7209523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7210088Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7210715Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7211169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7211637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7211991Z dist init r=1, world=2 2022-11-23T03:31:40.7212305Z dist init r=0, world=2 2022-11-23T03:31:40.7212528Z ok (6.538s) 2022-11-23T03:31:40.7213033Z test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7572 2022-11-23T03:31:40.7213629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7573 2022-11-23T03:31:40.7214266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7214708Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7215296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7215760Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7216197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7216888Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7217333Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7217920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7218378Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7218815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7219475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7220165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7220732Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7221233Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7221676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7222142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7222486Z dist init r=0, world=2 2022-11-23T03:31:40.7222728Z dist init r=1, world=2 2022-11-23T03:31:40.7222961Z ok (6.532s) 2022-11-23T03:31:40.7223456Z test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7725 2022-11-23T03:31:40.7224053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7726 2022-11-23T03:31:40.7224675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7225131Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7225719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7226182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7226621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7227245Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7227690Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7228282Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7228742Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7229186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7229913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7230617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7231191Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7231684Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7232126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7232590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7232943Z dist init r=1, world=2 2022-11-23T03:31:40.7233183Z dist init r=0, world=2 2022-11-23T03:31:40.7233413Z ok (6.638s) 2022-11-23T03:31:40.7233907Z test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7878 2022-11-23T03:31:40.7234553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7879 2022-11-23T03:31:40.7235179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7235618Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7236206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7236664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7237104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7237737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7238176Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7238762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7239223Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7239659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7240318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7241008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7241578Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7242069Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7242518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7242985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7243332Z dist init r=1, world=2 2022-11-23T03:31:40.7243574Z dist init r=0, world=2 2022-11-23T03:31:40.7243808Z ok (6.731s) 2022-11-23T03:31:40.7244316Z test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8031 2022-11-23T03:31:40.7244902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8032 2022-11-23T03:31:40.7245526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7245969Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7246558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7247089Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7247534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7248171Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7248603Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7249186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7249648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7250088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7250801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7251601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7252170Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7252671Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7253103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7253569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7253915Z dist init r=0, world=2 2022-11-23T03:31:40.7254157Z dist init r=1, world=2 2022-11-23T03:31:40.7254385Z ok (6.435s) 2022-11-23T03:31:40.7254771Z test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7255498Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8184 2022-11-23T03:31:40.7256033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8185 2022-11-23T03:31:40.7256646Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7257088Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7257672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7258134Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7258573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7259210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7259642Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7260233Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7260696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7261132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7261791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7262487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7263055Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7263549Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7263995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7264527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7264877Z dist init r=1, world=2 2022-11-23T03:31:40.7265124Z dist init r=0, world=2 2022-11-23T03:31:40.7265354Z ok (4.531s) 2022-11-23T03:31:40.7265730Z test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7266465Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8327 2022-11-23T03:31:40.7266995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8328 2022-11-23T03:31:40.7267613Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7268055Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7268642Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7269164Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7269605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7270231Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7270675Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7271260Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7271722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7272161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7272822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7273520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7274087Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7274578Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7275022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7275491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7275838Z dist init r=0, world=2 2022-11-23T03:31:40.7276078Z dist init r=1, world=2 2022-11-23T03:31:40.7276311Z ok (4.930s) 2022-11-23T03:31:40.7276693Z test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7277425Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8470 2022-11-23T03:31:40.7277957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8471 2022-11-23T03:31:40.7278575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7279017Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7279606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7280066Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7280497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7281136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7281576Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7282226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7282692Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7283134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7283795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7284486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7285047Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7285552Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7285995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7286467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7286867Z dist init r=0, world=2 2022-11-23T03:31:40.7287109Z dist init r=1, world=2 2022-11-23T03:31:40.7287331Z ok (4.431s) 2022-11-23T03:31:40.7287722Z test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7288458Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8613 2022-11-23T03:31:40.7288986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8614 2022-11-23T03:31:40.7289606Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7290048Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7290691Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7291166Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7291597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7292227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7292671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7293258Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7293718Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7294155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7294817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7295507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7296074Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7296574Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7297014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7297483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7297829Z dist init r=0, world=2 2022-11-23T03:31:40.7298070Z dist init r=1, world=2 2022-11-23T03:31:40.7298290Z ok (4.531s) 2022-11-23T03:31:40.7298675Z test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7299407Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8756 2022-11-23T03:31:40.7300006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8757 2022-11-23T03:31:40.7300636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7301077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7301666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7302117Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7302554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7303189Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7303635Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7304225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7304748Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7305186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7305837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7306526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7307099Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7307602Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7308048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7308514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7308864Z dist init r=0, world=2 2022-11-23T03:31:40.7309097Z dist init r=1, world=2 2022-11-23T03:31:40.7309326Z ok (4.831s) 2022-11-23T03:31:40.7309721Z test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) 2022-11-23T03:31:40.7310458Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8899 2022-11-23T03:31:40.7310980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8900 2022-11-23T03:31:40.7311597Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7312040Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7312617Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7313085Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7313525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:31:40.7314164Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:31:40.7314606Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:31:40.7315190Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:31:40.7315648Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:31:40.7316075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:31:40.7316733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7317484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:31:40.7318064Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7318567Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:31:40.7319011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:31:40.7319474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:31:40.7319824Z dist init r=1, world=2 2022-11-23T03:31:40.7320056Z dist init r=0, world=2 2022-11-23T03:31:40.7320288Z ok (4.632s) 2022-11-23T03:31:40.7320428Z 2022-11-23T03:31:40.7320702Z ---------------------------------------------------------------------- 2022-11-23T03:31:40.7321023Z Ran 27 tests in 127.228s 2022-11-23T03:31:40.7321179Z 2022-11-23T03:31:40.7321274Z OK (skipped=6) 2022-11-23T03:31:40.7321417Z 2022-11-23T03:31:40.7321530Z Generating XML reports... 2022-11-23T03:31:40.7322302Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20221123032931.xml 2022-11-23T03:31:40.7322653Z 2022-11-23T03:31:40.7323053Z ##[endgroup] 2022-11-23T03:31:40.7323680Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_wxc1jy08) 2022-11-23T03:31:40.7324033Z 2022-11-23T03:31:40.7324323Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-11-23 03:31:40.693872] 2022-11-23T03:31:40.7325047Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:31:40.694599] 2022-11-23T03:32:16.0734850Z 2022-11-23T03:32:16.0736591Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_clip_grad_norm 2022-11-23T03:32:16.0739167Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_aphv94mn) 2022-11-23T03:32:16.0741572Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm 2022-11-23T03:32:16.0742688Z 2022-11-23T03:32:16.0742978Z Running tests... 2022-11-23T03:32:16.0744470Z ---------------------------------------------------------------------- 2022-11-23T03:32:16.0745741Z test_ddp_parity (__main__.TestClipGradNorm) 2022-11-23T03:32:16.0747625Z Tests FSDP with ``FullyShardedDataParallel.clip_grad_norm_()` against ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9109 2022-11-23T03:32:16.0749490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9110 2022-11-23T03:32:16.0751841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:32:16.0753440Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:32:16.0755605Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:32:16.0757246Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:32:16.0758825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:32:16.0761209Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:32:16.0762609Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:32:16.0764873Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:32:16.0766731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:32:16.0768207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:32:16.0771630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:32:16.0774098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:32:16.0776007Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:32:16.0777492Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:32:16.0778764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:32:16.0780118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:32:16.0781486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0782862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0784231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0785928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0787297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0788667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0790037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0791398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0792790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0794161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0830712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0832092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0833377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0834636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0835885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0837142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0838406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0839644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0840897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0842142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0843419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0844669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0845910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0847159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0848394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0849639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0851284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0852561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0853841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0855437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0856723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0857970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0861098Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0864887Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0868848Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0872623Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0876389Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0878476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0879802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0881127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0882450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0883771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0885090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0886412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0887704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0889016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0890358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0892039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0893370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0894707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0896065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0897402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0898744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0900071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0900822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0901293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0901753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0902210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0902659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0903117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0903580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0904617Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0905942Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0907220Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0908495Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0909771Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0910520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0911066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0911612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0912159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0913362Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0914227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0914781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0915328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0915882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0916434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0916995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0917544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0918168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0918722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0919267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0919817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0920379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:32:16.0921409Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0922678Z [W python_variable.cpp:314] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function decref) 2022-11-23T03:32:16.0923324Z dist init r=0, world=2 2022-11-23T03:32:16.0923563Z dist init r=1, world=2 2022-11-23T03:32:16.0923790Z ok (24.780s) 2022-11-23T03:32:16.0924043Z test_non_root (__main__.TestClipGradNorm) 2022-11-23T03:32:16.0924684Z Tests that calling ``clip_grad_norm_()`` on a non-root FSDP instance ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9262 2022-11-23T03:32:16.0925206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9263 2022-11-23T03:32:16.0925823Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:32:16.0926261Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:32:16.0926853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:32:16.0927316Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:32:16.0927744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:32:16.0928376Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:32:16.0928813Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:32:16.0929395Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:32:16.0929853Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:32:16.0930284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:32:16.0931030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:32:16.0931819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:32:16.0932527Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:32:16.0933120Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:32:16.0933641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:32:16.0934193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:32:16.0934600Z dist init r=1, world=2 2022-11-23T03:32:16.0934881Z dist init r=0, world=2 2022-11-23T03:32:16.0935154Z ok (6.634s) 2022-11-23T03:32:16.0935317Z 2022-11-23T03:32:16.0935636Z ---------------------------------------------------------------------- 2022-11-23T03:32:16.0936010Z Ran 2 tests in 31.415s 2022-11-23T03:32:16.0936189Z 2022-11-23T03:32:16.0936280Z OK 2022-11-23T03:32:16.0936502Z 2022-11-23T03:32:16.0936632Z Generating XML reports... 2022-11-23T03:32:16.0937376Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20221123033142.xml 2022-11-23T03:32:16.0937785Z 2022-11-23T03:32:16.0938197Z ##[endgroup] 2022-11-23T03:32:16.0938954Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_aphv94mn) 2022-11-23T03:32:16.0939388Z 2022-11-23T03:32:16.0939706Z Running distributed/fsdp/test_fsdp_apply ... [2022-11-23 03:32:16.074148] 2022-11-23T03:32:16.0940534Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:32:16.074826] 2022-11-23T03:32:35.6772730Z 2022-11-23T03:32:35.6773855Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_apply 2022-11-23T03:32:35.6776857Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_apply_gck9fp96) 2022-11-23T03:32:35.6779083Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply 2022-11-23T03:32:35.6779804Z 2022-11-23T03:32:35.6780066Z Running tests... 2022-11-23T03:32:35.6781582Z ---------------------------------------------------------------------- 2022-11-23T03:32:35.6783002Z test_apply_in_summon_raises_error (__main__.TestApply) 2022-11-23T03:32:35.6784932Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9482 2022-11-23T03:32:35.6786857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9483 2022-11-23T03:32:35.6788950Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:32:35.6790141Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:32:35.6791701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:32:35.6792899Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:32:35.6794058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:32:35.6795744Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:32:35.6796893Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:32:35.6798424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:32:35.6799623Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:32:35.6800768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:32:35.6802509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:32:35.6804358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:32:35.6805864Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:32:35.6807446Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:32:35.6808591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:32:35.6809817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:32:35.6811061Z dist init r=1, world=2 2022-11-23T03:32:35.6814645Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:32:35.6816740Z warnings.warn( 2022-11-23T03:32:35.6817358Z dist init r=0, world=2 2022-11-23T03:32:35.6818172Z Asserting FSDP instance is: FullyShardedDataParallel( 2022-11-23T03:32:35.6819167Z (_fsdp_wrapped_module): TransformerWithSharedParams( 2022-11-23T03:32:35.6820002Z (embed_tokens): Embedding(23, 16) 2022-11-23T03:32:35.6820726Z (transformer): Transformer( 2022-11-23T03:32:35.6821455Z (encoder): TransformerEncoder( 2022-11-23T03:32:35.6822163Z (layers): ModuleList( 2022-11-23T03:32:35.6822891Z (0): FullyShardedDataParallel( 2022-11-23T03:32:35.6823780Z (_fsdp_wrapped_module): TransformerEncoderLayer( 2022-11-23T03:32:35.6824772Z (self_attn): MultiheadAttention( 2022-11-23T03:32:35.6825816Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-11-23T03:32:35.6826707Z ) 2022-11-23T03:32:35.6827468Z (linear1): Linear(in_features=16, out_features=8, bias=True) 2022-11-23T03:32:35.6828367Z (dropout): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6829269Z (linear2): Linear(in_features=8, out_features=16, bias=True) 2022-11-23T03:32:35.6830503Z (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6831718Z (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6832626Z (dropout1): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6833477Z (dropout2): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6834160Z ) 2022-11-23T03:32:35.6834701Z ) 2022-11-23T03:32:35.6835339Z (1): FullyShardedDataParallel( 2022-11-23T03:32:35.6836245Z (_fsdp_wrapped_module): TransformerEncoderLayer( 2022-11-23T03:32:35.6837115Z (self_attn): MultiheadAttention( 2022-11-23T03:32:35.6838152Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-11-23T03:32:35.6839036Z ) 2022-11-23T03:32:35.6839799Z (linear1): Linear(in_features=16, out_features=8, bias=True) 2022-11-23T03:32:35.6840679Z (dropout): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6841574Z (linear2): Linear(in_features=8, out_features=16, bias=True) 2022-11-23T03:32:35.6842789Z (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6843990Z (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6844886Z (dropout1): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6845736Z (dropout2): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6846415Z ) 2022-11-23T03:32:35.6846952Z ) 2022-11-23T03:32:35.6847483Z ) 2022-11-23T03:32:35.6848470Z (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6849203Z ) 2022-11-23T03:32:35.6849821Z (decoder): TransformerDecoder( 2022-11-23T03:32:35.6850514Z (layers): ModuleList( 2022-11-23T03:32:35.6851370Z (0): FullyShardedDataParallel( 2022-11-23T03:32:35.6852257Z (_fsdp_wrapped_module): TransformerDecoderLayer( 2022-11-23T03:32:35.6853129Z (self_attn): MultiheadAttention( 2022-11-23T03:32:35.6854175Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-11-23T03:32:35.6855059Z ) 2022-11-23T03:32:35.6855746Z (multihead_attn): MultiheadAttention( 2022-11-23T03:32:35.6856810Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-11-23T03:32:35.6857700Z ) 2022-11-23T03:32:35.6858633Z (linear1): Linear(in_features=16, out_features=8, bias=True) 2022-11-23T03:32:35.6859547Z (dropout): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6860454Z (linear2): Linear(in_features=8, out_features=16, bias=True) 2022-11-23T03:32:35.6861681Z (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6862881Z (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6864091Z (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6864987Z (dropout1): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6865832Z (dropout2): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6866674Z (dropout3): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6867337Z ) 2022-11-23T03:32:35.6867868Z ) 2022-11-23T03:32:35.6868668Z (1): FullyShardedDataParallel( 2022-11-23T03:32:35.6869564Z (_fsdp_wrapped_module): TransformerDecoderLayer( 2022-11-23T03:32:35.6870432Z (self_attn): MultiheadAttention( 2022-11-23T03:32:35.6871475Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-11-23T03:32:35.6872347Z ) 2022-11-23T03:32:35.6873044Z (multihead_attn): MultiheadAttention( 2022-11-23T03:32:35.6874102Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-11-23T03:32:35.6874983Z ) 2022-11-23T03:32:35.6875742Z (linear1): Linear(in_features=16, out_features=8, bias=True) 2022-11-23T03:32:35.6876637Z (dropout): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6877522Z (linear2): Linear(in_features=8, out_features=16, bias=True) 2022-11-23T03:32:35.6878767Z (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6879988Z (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6881189Z (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6882082Z (dropout1): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6882928Z (dropout2): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6883768Z (dropout3): Dropout(p=0.1, inplace=False) 2022-11-23T03:32:35.6884426Z ) 2022-11-23T03:32:35.6884959Z ) 2022-11-23T03:32:35.6885484Z ) 2022-11-23T03:32:35.6886467Z (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-11-23T03:32:35.6887210Z ) 2022-11-23T03:32:35.6887696Z ) 2022-11-23T03:32:35.6888433Z (output_proj): Linear(in_features=16, out_features=23, bias=True) 2022-11-23T03:32:35.6889756Z (bn): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) 2022-11-23T03:32:35.6890641Z ) 2022-11-23T03:32:35.6891135Z ) 2022-11-23T03:32:35.6892092Z ERROR: expected to be in states [] but current state is TrainingState.SUMMON_FULL_PARAMS 2022-11-23T03:32:35.6895708Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:32:35.6897756Z warnings.warn( 2022-11-23T03:32:35.6898403Z File "", line 1, in 2022-11-23T03:32:35.6899344Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main 2022-11-23T03:32:35.6900290Z exitcode = _main(fd, parent_sentinel) 2022-11-23T03:32:35.6901233Z File "/opt/conda/lib/python3.8/multiprocessing/spawn.py", line 129, in _main 2022-11-23T03:32:35.6902354Z return self._bootstrap(parent_sentinel) 2022-11-23T03:32:35.6903371Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap 2022-11-23T03:32:35.6904203Z self.run() 2022-11-23T03:32:35.6905064Z File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run 2022-11-23T03:32:35.6905998Z self._target(*self._args, **self._kwargs) 2022-11-23T03:32:35.6907385Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_fsdp.py", line 790, in _run 2022-11-23T03:32:35.6908375Z self.run_test(test_name, pipe) 2022-11-23T03:32:35.6909806Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 656, in run_test 2022-11-23T03:32:35.6910800Z getattr(self, test_name)() 2022-11-23T03:32:35.6912204Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 534, in wrapper 2022-11-23T03:32:35.6913299Z fn() 2022-11-23T03:32:35.6914613Z File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 166, in wrapper 2022-11-23T03:32:35.6915621Z return func(*args, **kwargs) 2022-11-23T03:32:35.6916684Z File "/var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_apply.py", line 98, in test_apply_in_summon_raises_error 2022-11-23T03:32:35.6917803Z transformer.apply(self._init_linear_weights) 2022-11-23T03:32:35.6919307Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 492, in apply 2022-11-23T03:32:35.6920411Z self._assert_state(TrainingState.IDLE) 2022-11-23T03:32:35.6921950Z File "/opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1084, in _assert_state 2022-11-23T03:32:35.6923002Z traceback.print_stack() 2022-11-23T03:32:35.6923636Z ok (5.232s) 2022-11-23T03:32:35.6924335Z test_nested_module_apply (__main__.TestApply) 2022-11-23T03:32:35.6926015Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9625 2022-11-23T03:32:35.6927403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9626 2022-11-23T03:32:35.6929039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:32:35.6930191Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:32:35.6932048Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:32:35.6933258Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:32:35.6934404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:32:35.6936077Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:32:35.6937238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:32:35.6938765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:32:35.6939960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:32:35.6941104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:32:35.6942519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:32:35.6943299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:32:35.6943870Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:32:35.6944357Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:32:35.6944864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:32:35.6945326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:32:35.6945669Z dist init r=0, world=2 2022-11-23T03:32:35.6946884Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:32:35.6947655Z warnings.warn( 2022-11-23T03:32:35.6947890Z dist init r=1, world=2 2022-11-23T03:32:35.6949094Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:32:35.6949920Z warnings.warn( 2022-11-23T03:32:35.6950141Z ok (4.930s) 2022-11-23T03:32:35.6950411Z test_transformer_module_apply (__main__.TestApply) 2022-11-23T03:32:35.6951057Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9772 2022-11-23T03:32:35.6951582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9773 2022-11-23T03:32:35.6952192Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:32:35.6952626Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:32:35.6953210Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:32:35.6953663Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:32:35.6954086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:32:35.6954713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:32:35.6955145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:32:35.6955723Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:32:35.6956173Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:32:35.6956605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:32:35.6957260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:32:35.6957954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:32:35.6958513Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:32:35.6959008Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:32:35.6959448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:32:35.6959903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:32:35.6960247Z dist init r=0, world=2 2022-11-23T03:32:35.6961502Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:32:35.6962268Z warnings.warn( 2022-11-23T03:32:35.6962502Z dist init r=1, world=2 2022-11-23T03:32:35.6963702Z /opt/conda/lib/python3.8/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication. 2022-11-23T03:32:35.6964456Z warnings.warn( 2022-11-23T03:32:35.6964678Z ok (5.333s) 2022-11-23T03:32:35.6964814Z 2022-11-23T03:32:35.6965086Z ---------------------------------------------------------------------- 2022-11-23T03:32:35.6965450Z Ran 3 tests in 15.496s 2022-11-23T03:32:35.6965601Z 2022-11-23T03:32:35.6965678Z OK 2022-11-23T03:32:35.6965797Z 2022-11-23T03:32:35.6965904Z Generating XML reports... 2022-11-23T03:32:35.6966478Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221123033217.xml 2022-11-23T03:32:35.6966789Z 2022-11-23T03:32:35.6967075Z ##[endgroup] 2022-11-23T03:32:35.6967673Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_fsdp_apply_gck9fp96) 2022-11-23T03:32:35.6968013Z 2022-11-23T03:32:35.6968297Z Running distributed/fsdp/test_checkpoint_wrapper ... [2022-11-23 03:32:35.677873] 2022-11-23T03:32:35.6969016Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:32:35.678604] 2022-11-23T03:32:42.5485263Z 2022-11-23T03:32:42.5486393Z Expand the folded group to see the log file of distributed/fsdp/test_checkpoint_wrapper 2022-11-23T03:32:42.5489215Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_j4i5pu84) 2022-11-23T03:32:42.5491630Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper 2022-11-23T03:32:42.5492392Z 2022-11-23T03:32:42.5492634Z Running tests... 2022-11-23T03:32:42.5493708Z ---------------------------------------------------------------------- 2022-11-23T03:32:42.5495167Z test_apply_activation_checkpointing (__main__.CheckpointWrapperTest) 2022-11-23T03:32:42.5496498Z Ensures that `apply_activation_checkpointing` can be used ... ok (0.713s) 2022-11-23T03:32:42.5497863Z test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... ok (1.571s) 2022-11-23T03:32:42.5499349Z test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... ok (0.009s) 2022-11-23T03:32:42.5500502Z test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest) 2022-11-23T03:32:42.5503534Z Tests that using checkpoint_wrapper or the functional ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79510 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.003s) 2022-11-23T03:32:42.5505584Z test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... ok (0.001s) 2022-11-23T03:32:42.5506628Z test_fqn (__main__.CheckpointWrapperTest) ... ok (0.001s) 2022-11-23T03:32:42.5507708Z test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... ok (0.003s) 2022-11-23T03:32:42.5508371Z 2022-11-23T03:32:42.5509066Z ---------------------------------------------------------------------- 2022-11-23T03:32:42.5509897Z Ran 7 tests in 2.301s 2022-11-23T03:32:42.5510285Z 2022-11-23T03:32:42.5510527Z OK (skipped=1) 2022-11-23T03:32:42.5510890Z 2022-11-23T03:32:42.5511611Z Generating XML reports... 2022-11-23T03:32:42.5513149Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221123033237.xml 2022-11-23T03:32:42.5513951Z 2022-11-23T03:32:42.5514552Z ##[endgroup] 2022-11-23T03:32:42.5515998Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/pytorch/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_j4i5pu84) 2022-11-23T03:32:42.5516810Z 2022-11-23T03:32:42.5517458Z Running distributed/elastic/utils/logging_test ... [2022-11-23 03:32:42.548800] 2022-11-23T03:32:42.5519032Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:32:42.549496] 2022-11-23T03:32:47.1539892Z 2022-11-23T03:32:47.1541225Z Expand the folded group to see the log file of distributed/elastic/utils/logging_test 2022-11-23T03:32:47.1544848Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-logging_test_0ghhk1l_) 2022-11-23T03:32:47.1546975Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test 2022-11-23T03:32:47.1547800Z 2022-11-23T03:32:47.1548219Z Running tests... 2022-11-23T03:32:47.1549680Z ---------------------------------------------------------------------- 2022-11-23T03:32:47.1550702Z test_derive_module_name (__main__.LoggingTest) ... ok (0.626s) 2022-11-23T03:32:47.1551622Z test_logger_name (__main__.LoggingTest) ... ok (0.002s) 2022-11-23T03:32:47.1552129Z 2022-11-23T03:32:47.1553064Z ---------------------------------------------------------------------- 2022-11-23T03:32:47.1554128Z Ran 2 tests in 0.629s 2022-11-23T03:32:47.1554632Z 2022-11-23T03:32:47.1554862Z OK 2022-11-23T03:32:47.1555177Z 2022-11-23T03:32:47.1555465Z Generating XML reports... 2022-11-23T03:32:47.1557117Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221123033244.xml 2022-11-23T03:32:47.1558003Z 2022-11-23T03:32:47.1558721Z ##[endgroup] 2022-11-23T03:32:47.1560414Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-utils-logging_test_0ghhk1l_) 2022-11-23T03:32:47.1561381Z 2022-11-23T03:32:47.1562150Z Running distributed/elastic/timer/local_timer_test ... [2022-11-23 03:32:47.154400] 2022-11-23T03:32:47.1564084Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:32:47.155041] 2022-11-23T03:32:56.0471682Z 2022-11-23T03:32:56.0473074Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_test 2022-11-23T03:32:56.0476110Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-local_timer_test_jdx4cw0m) 2022-11-23T03:32:56.0478480Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test 2022-11-23T03:32:56.0479253Z 2022-11-23T03:32:56.0479505Z Running tests... 2022-11-23T03:32:56.0480606Z ---------------------------------------------------------------------- 2022-11-23T03:32:56.0481598Z test_acquire_release (__main__.LocalTimerServerTest) 2022-11-23T03:32:56.0484319Z tests that: ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/87154 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.596s) 2022-11-23T03:32:56.0486137Z test_expired_timers (__main__.LocalTimerServerTest) 2022-11-23T03:32:56.0487168Z tests that a single expired timer on a process should terminate ... ok (0.004s) 2022-11-23T03:32:56.0488674Z test_valid_timers (__main__.LocalTimerServerTest) 2022-11-23T03:32:56.0489764Z tests that valid timers are processed correctly and the process is left alone ... ok (0.004s) 2022-11-23T03:32:56.0491006Z test_watchdog_call_count (__main__.LocalTimerServerTest) 2022-11-23T03:32:56.0492341Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.105s) 2022-11-23T03:32:56.0493359Z test_watchdog_empty_queue (__main__.LocalTimerServerTest) 2022-11-23T03:32:56.0494359Z checks that the watchdog can run on an empty queue ... ok (0.012s) 2022-11-23T03:32:56.0495355Z test_client_interaction (__main__.LocalTimerTest) ... ok (0.016s) 2022-11-23T03:32:56.0496370Z test_exception_propagation (__main__.LocalTimerTest) ... ok (0.012s) 2022-11-23T03:32:56.0497331Z test_get_timer_recursive (__main__.LocalTimerTest) 2022-11-23T03:32:56.0498328Z If a function acquires a countdown timer with default scope, ... ok (2.318s) 2022-11-23T03:32:56.0499338Z test_happy_path (__main__.LocalTimerTest) ... ok (0.107s) 2022-11-23T03:32:56.0500587Z test_no_client (__main__.LocalTimerTest) ... ok (0.013s) 2022-11-23T03:32:56.0501469Z test_timer (__main__.LocalTimerTest) ... ok (0.150s) 2022-11-23T03:32:56.0502453Z test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.025s) 2022-11-23T03:32:56.0503543Z test_get_less_than_size (__main__.MultiprocessingRequestQueueTest) 2022-11-23T03:32:56.0504461Z Tests slow producer. ... ok (0.515s) 2022-11-23T03:32:56.0505360Z test_get_size (__main__.MultiprocessingRequestQueueTest) 2022-11-23T03:32:56.0506376Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.918s) 2022-11-23T03:32:56.0506963Z 2022-11-23T03:32:56.0507709Z ---------------------------------------------------------------------- 2022-11-23T03:32:56.0508555Z Ran 14 tests in 4.803s 2022-11-23T03:32:56.0508956Z 2022-11-23T03:32:56.0509202Z OK (skipped=1) 2022-11-23T03:32:56.0509572Z 2022-11-23T03:32:56.0509901Z Generating XML reports... 2022-11-23T03:32:56.0511619Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20221123033248.xml 2022-11-23T03:32:56.0513758Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20221123033248.xml 2022-11-23T03:32:56.0516015Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20221123033248.xml 2022-11-23T03:32:56.0517047Z 2022-11-23T03:32:56.0517756Z ##[endgroup] 2022-11-23T03:32:56.0519475Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-local_timer_test_jdx4cw0m) 2022-11-23T03:32:56.0520445Z 2022-11-23T03:32:56.0521164Z Running distributed/elastic/timer/api_test ... [2022-11-23 03:32:56.047490] 2022-11-23T03:32:56.0523018Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:32:56.048161] 2022-11-23T03:33:00.0885927Z 2022-11-23T03:33:00.0886977Z Expand the folded group to see the log file of distributed/elastic/timer/api_test 2022-11-23T03:33:00.0889383Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-api_test_lycnz9wl) 2022-11-23T03:33:00.0890341Z 2022-11-23T03:33:00.0891373Z ##[endgroup] 2022-11-23T03:33:00.0893234Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-timer-api_test_lycnz9wl) 2022-11-23T03:33:00.0894536Z 2022-11-23T03:33:00.0895590Z Running distributed/elastic/metrics/api_test ... [2022-11-23 03:33:00.088835] 2022-11-23T03:33:00.0899603Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:33:00.089527] 2022-11-23T03:33:05.1311731Z 2022-11-23T03:33:05.1313519Z Expand the folded group to see the log file of distributed/elastic/metrics/api_test 2022-11-23T03:33:05.1316492Z ##[group]PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-metrics-api_test_g170fg42) 2022-11-23T03:33:05.1318530Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test 2022-11-23T03:33:05.1319284Z 2022-11-23T03:33:05.1319535Z Running tests... 2022-11-23T03:33:05.1320957Z ---------------------------------------------------------------------- 2022-11-23T03:33:05.1322287Z test_get_metric_name (__main__.MetricsApiTest) ... ok (0.587s) 2022-11-23T03:33:05.1323257Z test_inheritance (__main__.MetricsApiTest) ... ok (0.002s) 2022-11-23T03:33:05.1324177Z test_profile (__main__.MetricsApiTest) ... ok (0.002s) 2022-11-23T03:33:05.1324950Z 2022-11-23T03:33:05.1325693Z ---------------------------------------------------------------------- 2022-11-23T03:33:05.1326834Z Ran 3 tests in 0.591s 2022-11-23T03:33:05.1327226Z 2022-11-23T03:33:05.1327437Z OK 2022-11-23T03:33:05.1327751Z 2022-11-23T03:33:05.1328036Z Generating XML reports... 2022-11-23T03:33:05.1329673Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20221123033301.xml 2022-11-23T03:33:05.1330700Z 2022-11-23T03:33:05.1331402Z ##[endgroup] 2022-11-23T03:33:05.1333083Z FINISHED PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/pytorch/test/test-reports/distributed-elastic-metrics-api_test_g170fg42) 2022-11-23T03:33:05.1334035Z 2022-11-23T03:33:05.1334778Z Running distributed/checkpoint/test_utils ... [2022-11-23 03:33:05.131472] 2022-11-23T03:33:05.1336640Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:33:05.132086] 2022-11-23T03:33:09.6751442Z 2022-11-23T03:33:09.6752370Z Expand the folded group to see the log file of distributed/checkpoint/test_utils 2022-11-23T03:33:09.6754947Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_utils_5g810t6e) 2022-11-23T03:33:09.6757123Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_utils 2022-11-23T03:33:09.6757858Z 2022-11-23T03:33:09.6758108Z Running tests... 2022-11-23T03:33:09.6759210Z ---------------------------------------------------------------------- 2022-11-23T03:33:09.6760239Z test_flat_data (__main__.TestMedatadaIndex) ... ok (0.602s) 2022-11-23T03:33:09.6761383Z test_index_hint_ignored_on_equals (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-11-23T03:33:09.6762546Z test_index_hint_ignored_on_hash (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-11-23T03:33:09.6763586Z test_init_convert_offset (__main__.TestMedatadaIndex) ... ok (0.001s) 2022-11-23T03:33:09.6764619Z test_sharded_tensor_lookup (__main__.TestMedatadaIndex) ... ok (0.003s) 2022-11-23T03:33:09.6765276Z 2022-11-23T03:33:09.6766026Z ---------------------------------------------------------------------- 2022-11-23T03:33:09.6766855Z Ran 5 tests in 0.609s 2022-11-23T03:33:09.6767250Z 2022-11-23T03:33:09.6767433Z OK 2022-11-23T03:33:09.6767740Z 2022-11-23T03:33:09.6768026Z Generating XML reports... 2022-11-23T03:33:09.6769626Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_utils/TEST-TestMedatadaIndex-20221123033306.xml 2022-11-23T03:33:09.6770729Z 2022-11-23T03:33:09.6771449Z ##[endgroup] 2022-11-23T03:33:09.6773075Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_utils (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_utils_5g810t6e) 2022-11-23T03:33:09.6773995Z 2022-11-23T03:33:09.6774723Z Running distributed/checkpoint/test_planner ... [2022-11-23 03:33:09.675460] 2022-11-23T03:33:09.6777061Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:33:09.676117] 2022-11-23T03:33:14.2805361Z 2022-11-23T03:33:14.2806390Z Expand the folded group to see the log file of distributed/checkpoint/test_planner 2022-11-23T03:33:14.2809420Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_planner_ayy7c9en) 2022-11-23T03:33:14.2812396Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_planner 2022-11-23T03:33:14.2813292Z 2022-11-23T03:33:14.2813560Z Running tests... 2022-11-23T03:33:14.2814686Z ---------------------------------------------------------------------- 2022-11-23T03:33:14.2815708Z test_global_plan (__main__.TestSavePlan) ... ok (0.603s) 2022-11-23T03:33:14.2816694Z test_load_with_resharding (__main__.TestSavePlan) ... ok (0.005s) 2022-11-23T03:33:14.2817746Z test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... ok (0.003s) 2022-11-23T03:33:14.2819259Z test_local_load_plan (__main__.TestSavePlan) ... ok (0.004s) 2022-11-23T03:33:14.2820261Z test_local_plan (__main__.TestSavePlan) ... ok (0.004s) 2022-11-23T03:33:14.2820796Z 2022-11-23T03:33:14.2821559Z ---------------------------------------------------------------------- 2022-11-23T03:33:14.2822407Z Ran 5 tests in 0.619s 2022-11-23T03:33:14.2822808Z 2022-11-23T03:33:14.2823023Z OK 2022-11-23T03:33:14.2823356Z 2022-11-23T03:33:14.2823657Z Generating XML reports... 2022-11-23T03:33:14.2825276Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20221123033311.xml 2022-11-23T03:33:14.2826172Z 2022-11-23T03:33:14.2826909Z ##[endgroup] 2022-11-23T03:33:14.2828609Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_planner_ayy7c9en) 2022-11-23T03:33:14.2829572Z 2022-11-23T03:33:14.2830430Z Running distributed/checkpoint/test_file_system_checkpoint ... [2022-11-23 03:33:14.280759] 2022-11-23T03:33:14.2832520Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:33:14.281475] 2022-11-23T03:33:35.8730924Z 2022-11-23T03:33:35.8732615Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint 2022-11-23T03:33:35.8740951Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_bwuk9ydj) 2022-11-23T03:33:35.8743421Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint 2022-11-23T03:33:35.8744308Z 2022-11-23T03:33:35.8744571Z Running tests... 2022-11-23T03:33:35.8745747Z ---------------------------------------------------------------------- 2022-11-23T03:33:35.8749068Z test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84440 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.588s) 2022-11-23T03:33:35.8753465Z test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84531 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.003s) 2022-11-23T03:33:35.8756043Z test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10571 2022-11-23T03:33:35.8757539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10572 2022-11-23T03:33:35.8759733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:33:35.8761137Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:33:35.8762817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:33:35.8764101Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:33:35.8765304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:33:35.8766627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:33:35.8768424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:33:35.8769651Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:33:35.8771471Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:33:35.8772979Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:33:35.8774178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:33:35.8775495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:33:35.8777374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:33:35.8779320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:33:35.8780924Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:33:35.8782322Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:33:35.8783182Z ok (4.737s) 2022-11-23T03:33:35.8784492Z test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10716 2022-11-23T03:33:35.8786063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10717 2022-11-23T03:33:35.8787802Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:33:35.8789042Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:33:35.8790680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:33:35.8791964Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:33:35.8793161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:33:35.8794490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:33:35.8796267Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:33:35.8797494Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:33:35.8799136Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:33:35.8800417Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:33:35.8801610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:33:35.8802937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:33:35.8804790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:33:35.8806730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:33:35.8808475Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:33:35.8809894Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:33:35.8810883Z ok (6.537s) 2022-11-23T03:33:35.8811850Z test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.050s) 2022-11-23T03:33:35.8813608Z test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10863 2022-11-23T03:33:35.8815293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10864 2022-11-23T03:33:35.8817039Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:33:35.8818251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:33:35.8819896Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:33:35.8821371Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:33:35.8822572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:33:35.8823903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:33:35.8825695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:33:35.8826925Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:33:35.8828549Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:33:35.8829837Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:33:35.8831030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:33:35.8832362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:33:35.8834213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:33:35.8836149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:33:35.8837745Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:33:35.8839146Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:33:35.8839997Z ok (5.329s) 2022-11-23T03:33:35.8840371Z 2022-11-23T03:33:35.8841126Z ---------------------------------------------------------------------- 2022-11-23T03:33:35.8841999Z Ran 6 tests in 17.247s 2022-11-23T03:33:35.8842416Z 2022-11-23T03:33:35.8842667Z OK (skipped=2) 2022-11-23T03:33:35.8843057Z 2022-11-23T03:33:35.8843355Z Generating XML reports... 2022-11-23T03:33:35.8845278Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221123033316.xml 2022-11-23T03:33:35.8847872Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221123033316.xml 2022-11-23T03:33:35.8850948Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221123033316.xml 2022-11-23T03:33:35.8852240Z 2022-11-23T03:33:35.8852995Z ##[endgroup] 2022-11-23T03:33:35.8854928Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_bwuk9ydj) 2022-11-23T03:33:35.8856033Z 2022-11-23T03:33:35.8856832Z Running distributed/checkpoint/test_checkpoint ... [2022-11-23 03:33:35.873462] 2022-11-23T03:33:35.8859028Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:33:35.874138] 2022-11-23T03:34:15.7035005Z 2022-11-23T03:34:15.7035926Z Expand the folded group to see the log file of distributed/checkpoint/test_checkpoint 2022-11-23T03:34:15.7041316Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_checkpoint_gw5eu5dk) 2022-11-23T03:34:15.7043685Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_checkpoint 2022-11-23T03:34:15.7044506Z 2022-11-23T03:34:15.7044760Z Running tests... 2022-11-23T03:34:15.7045903Z ---------------------------------------------------------------------- 2022-11-23T03:34:15.7047393Z test_default_metadata (__main__.TestDistributedCheckpointing) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11077 2022-11-23T03:34:15.7048915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11078 2022-11-23T03:34:15.7052341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7053583Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7055206Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7056674Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7057884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:15.7059176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:34:15.7060959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7062165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7063805Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7065086Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7066268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:15.7067554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:34:15.7069380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:34:15.7071303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:34:15.7072866Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:15.7074237Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:15.7075096Z ok (5.717s) 2022-11-23T03:34:15.7076374Z test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11220 2022-11-23T03:34:15.7077919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11221 2022-11-23T03:34:15.7079622Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7080805Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7082409Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7083664Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7084842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:15.7086135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:34:15.7088176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7089383Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7091234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7092530Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7093716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:15.7095002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:34:15.7096856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:34:15.7098779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:34:15.7100568Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:15.7101943Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:15.7102787Z ok (4.329s) 2022-11-23T03:34:15.7103957Z test_dummy_reader_works (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11363 2022-11-23T03:34:15.7105404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11364 2022-11-23T03:34:15.7106624Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11365 2022-11-23T03:34:15.7107848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11366 2022-11-23T03:34:15.7109587Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7110818Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7112442Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7113731Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7114916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:15.7116666Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7117892Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7119512Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7120791Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7121965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:15.7123752Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7124968Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7126586Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7127848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7129034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:34:15.7130886Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7132091Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7133715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7134997Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7136347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:34:15.7137426Z skip: Need at least 4 CUDA devices (4.340s) 2022-11-23T03:34:15.7138719Z test_dummy_writer_works (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11631 2022-11-23T03:34:15.7140164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11632 2022-11-23T03:34:15.7141367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11633 2022-11-23T03:34:15.7142593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11634 2022-11-23T03:34:15.7144305Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7145516Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7147135Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7148526Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7149701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:15.7151460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7152653Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7154265Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7155512Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7156664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:34:15.7158399Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7159624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7161221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7162471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7163633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:34:15.7165539Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7166749Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7168341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7169588Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7171195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:15.7172420Z skip: Need at least 4 CUDA devices (4.441s) 2022-11-23T03:34:15.7173720Z test_load_error_handling (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11899 2022-11-23T03:34:15.7175169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11900 2022-11-23T03:34:15.7176387Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11901 2022-11-23T03:34:15.7177608Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11902 2022-11-23T03:34:15.7179362Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7180564Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7181633Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7182243Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7182669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:34:15.7183303Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7183745Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7184329Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7184792Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7185219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:15.7185847Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7186348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7186934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7187394Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7187821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:34:15.7188450Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7188890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7189472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7189934Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7190356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:15.7190746Z skip: Need at least 4 CUDA devices (4.045s) 2022-11-23T03:34:15.7191227Z test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12167 2022-11-23T03:34:15.7191761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12168 2022-11-23T03:34:15.7192201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12169 2022-11-23T03:34:15.7192643Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12170 2022-11-23T03:34:15.7193261Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7193689Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7194269Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7194734Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7195163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:15.7195784Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7196227Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7196816Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7197276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7197697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:34:15.7198318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7198756Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7199394Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7199854Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7200280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:34:15.7200910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7201338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7201919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7202374Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7202802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:15.7203187Z ok (4.348s) 2022-11-23T03:34:15.7203615Z test_save_error_handling (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12435 2022-11-23T03:34:15.7204142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12436 2022-11-23T03:34:15.7204577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12437 2022-11-23T03:34:15.7205017Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12438 2022-11-23T03:34:15.7205636Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7206076Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7206659Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7207115Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7207546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:15.7208166Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7208602Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7209186Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7209643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7210072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:34:15.7210747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7211188Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7211869Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7212426Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7212945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:34:15.7213695Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7214221Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7214917Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7215471Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7215980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:15.7216438Z skip: Need at least 4 CUDA devices (4.143s) 2022-11-23T03:34:15.7217081Z test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12703 2022-11-23T03:34:15.7217738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12704 2022-11-23T03:34:15.7218270Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12705 2022-11-23T03:34:15.7218796Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12706 2022-11-23T03:34:15.7219548Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7220070Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7220759Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7221317Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7221747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:34:15.7222504Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7222939Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7223520Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7223974Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7224389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:15.7225012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7225446Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7226030Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7226493Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7226918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:34:15.7227536Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:15.7227974Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:15.7228547Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:15.7228999Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:15.7229422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:15.7229753Z ok (4.542s) 2022-11-23T03:34:15.7229888Z 2022-11-23T03:34:15.7230160Z ---------------------------------------------------------------------- 2022-11-23T03:34:15.7230483Z Ran 8 tests in 35.906s 2022-11-23T03:34:15.7230635Z 2022-11-23T03:34:15.7230723Z OK (skipped=4) 2022-11-23T03:34:15.7230864Z 2022-11-23T03:34:15.7230974Z Generating XML reports... 2022-11-23T03:34:15.7231632Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20221123033337.xml 2022-11-23T03:34:15.7232476Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20221123033337.xml 2022-11-23T03:34:15.7232830Z 2022-11-23T03:34:15.7233133Z ##[endgroup] 2022-11-23T03:34:15.7233766Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/pytorch/test/test-reports/distributed-checkpoint-test_checkpoint_gw5eu5dk) 2022-11-23T03:34:15.7234127Z 2022-11-23T03:34:15.7234440Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2022-11-23 03:34:15.704248] 2022-11-23T03:34:15.7235253Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:34:15.704966] 2022-11-23T03:34:52.0953128Z 2022-11-23T03:34:52.0954428Z Expand the folded group to see the log file of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 2022-11-23T03:34:52.0956751Z ##[group]PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_dnvqlz6g) 2022-11-23T03:34:52.0959344Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks 2022-11-23T03:34:52.0960192Z 2022-11-23T03:34:52.0960445Z Running tests... 2022-11-23T03:34:52.0961553Z ---------------------------------------------------------------------- 2022-11-23T03:34:52.0962711Z test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest) 2022-11-23T03:34:52.0964272Z This unit test verifies the ``allreduce`` hook registered case gives same result ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13038 2022-11-23T03:34:52.0967069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13039 2022-11-23T03:34:52.0968942Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.0970447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.0972205Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.0973513Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.0975235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:52.0977859Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.0979078Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.0980716Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.0981966Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.0983100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:52.0984396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwim1ae7r 2022-11-23T03:34:52.0985770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwim1ae7r/_remote_module_non_scriptable.py 2022-11-23T03:34:52.0987107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24x7llig 2022-11-23T03:34:52.0988460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24x7llig/_remote_module_non_scriptable.py 2022-11-23T03:34:52.0989995Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.0991351Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.0992163Z ok (5.236s) 2022-11-23T03:34:52.0993084Z test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest) 2022-11-23T03:34:52.0994534Z This unit test verifies the ``fp16 compress`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13189 2022-11-23T03:34:52.0995929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13190 2022-11-23T03:34:52.0997569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.0998732Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.1000286Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.1001503Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.1002964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:52.1004662Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.1005823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.1007365Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.1021185Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.1022766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:52.1024676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxxje424 2022-11-23T03:34:52.1026549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxxje424/_remote_module_non_scriptable.py 2022-11-23T03:34:52.1028221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48popv89 2022-11-23T03:34:52.1029940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48popv89/_remote_module_non_scriptable.py 2022-11-23T03:34:52.1031596Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.1032930Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.1033739Z ok (4.529s) 2022-11-23T03:34:52.1034641Z test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest) 2022-11-23T03:34:52.1036125Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13340 2022-11-23T03:34:52.1037589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13341 2022-11-23T03:34:52.1039273Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.1040451Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.1042011Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.1043228Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.1044360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:52.1046025Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.1047167Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.1048712Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.1049928Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.1051703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:52.1053018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7ax1qcp 2022-11-23T03:34:52.1054381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7ax1qcp/_remote_module_non_scriptable.py 2022-11-23T03:34:52.1055724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbl89klf 2022-11-23T03:34:52.1057052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbl89klf/_remote_module_non_scriptable.py 2022-11-23T03:34:52.1058556Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.1059865Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.1060670Z ok (4.530s) 2022-11-23T03:34:52.1061612Z test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest) 2022-11-23T03:34:52.1062912Z This unit test verifies the ``quantize per channel`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13491 2022-11-23T03:34:52.1063944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13492 2022-11-23T03:34:52.1065118Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.1065945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.1067060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.1067925Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.1068725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:52.1069912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.1070734Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.1071961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.1072809Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.1073616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:52.1074527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxw5asduj 2022-11-23T03:34:52.1075498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxw5asduj/_remote_module_non_scriptable.py 2022-11-23T03:34:52.1076468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3bld6li 2022-11-23T03:34:52.1077434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3bld6li/_remote_module_non_scriptable.py 2022-11-23T03:34:52.1078487Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.1079419Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.1080002Z ok (4.628s) 2022-11-23T03:34:52.1080671Z test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest) 2022-11-23T03:34:52.1081743Z This unit test verifies the ``quantize per tensor`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13642 2022-11-23T03:34:52.1082749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13643 2022-11-23T03:34:52.1083920Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.1084743Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.1085834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.1086695Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.1087508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:52.1088700Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.1089521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.1090765Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.1091630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.1092425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:52.1093341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mda6avz 2022-11-23T03:34:52.1094307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mda6avz/_remote_module_non_scriptable.py 2022-11-23T03:34:52.1095271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjc0vnf2n 2022-11-23T03:34:52.1096377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjc0vnf2n/_remote_module_non_scriptable.py 2022-11-23T03:34:52.1097466Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.1098409Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.1098978Z ok (4.630s) 2022-11-23T03:34:52.1099821Z test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13793 2022-11-23T03:34:52.1100847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13794 2022-11-23T03:34:52.1102028Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.1102851Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.1103959Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.1104929Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.1105733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:34:52.1106912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:34:52.1107736Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:34:52.1108840Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:34:52.1109708Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:34:52.1110514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:34:52.1111428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2t13lbt 2022-11-23T03:34:52.1112402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2t13lbt/_remote_module_non_scriptable.py 2022-11-23T03:34:52.1113346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp503w9798 2022-11-23T03:34:52.1114301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp503w9798/_remote_module_non_scriptable.py 2022-11-23T03:34:52.1115337Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.1116282Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:34:52.1116860Z ok (8.845s) 2022-11-23T03:34:52.1117111Z 2022-11-23T03:34:52.1117627Z ---------------------------------------------------------------------- 2022-11-23T03:34:52.1118215Z Ran 6 tests in 32.400s 2022-11-23T03:34:52.1118485Z 2022-11-23T03:34:52.1118632Z OK 2022-11-23T03:34:52.1118851Z 2022-11-23T03:34:52.1119053Z Generating XML reports... 2022-11-23T03:34:52.1120409Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20221123033417.xml 2022-11-23T03:34:52.1121207Z 2022-11-23T03:34:52.1121883Z ##[endgroup] 2022-11-23T03:34:52.1123208Z FINISHED PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/pytorch/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_dnvqlz6g) 2022-11-23T03:34:52.1123957Z 2022-11-23T03:34:52.1124499Z Running distributed/_tensor/test_tp_sharding_ops ... [2022-11-23 03:34:52.095836] 2022-11-23T03:34:52.1125865Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tp_sharding_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:34:52.096498] 2022-11-23T03:35:22.8075920Z 2022-11-23T03:35:22.8077464Z Expand the folded group to see the log file of distributed/_tensor/test_tp_sharding_ops 2022-11-23T03:35:22.8082928Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_tp_sharding_ops_iw8hjxwj) 2022-11-23T03:35:22.8086005Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops 2022-11-23T03:35:22.8086835Z 2022-11-23T03:35:22.8087156Z Running tests... 2022-11-23T03:35:22.8088662Z ---------------------------------------------------------------------- 2022-11-23T03:35:22.8091030Z test_replicated_permute (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14013 2022-11-23T03:35:22.8092704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14014 2022-11-23T03:35:22.8094038Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14015 2022-11-23T03:35:22.8095312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14016 2022-11-23T03:35:22.8097208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8099447Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8101270Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8102770Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8104248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:35:22.8106509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8108153Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8110370Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8111890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8113137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:35:22.8114991Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8116270Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8117962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8119281Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8120507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:35:22.8122341Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8123613Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8125318Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8126820Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8128052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:35:22.8129167Z skip: Need at least 4 CUDA devices (4.738s) 2022-11-23T03:35:22.8130468Z test_sharded_cat (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14281 2022-11-23T03:35:22.8132234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14282 2022-11-23T03:35:22.8133404Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14283 2022-11-23T03:35:22.8134578Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14284 2022-11-23T03:35:22.8136308Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8137485Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8139324Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8140559Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8141690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:35:22.8143381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8144454Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8145599Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8146485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8147300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:35:22.8148620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8149475Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8150595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8151479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8152297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:35:22.8153508Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8154338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8155472Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8156344Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8157168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:35:22.8157887Z skip: Need at least 4 CUDA devices (4.949s) 2022-11-23T03:35:22.8158771Z test_sharded_permute (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14549 2022-11-23T03:35:22.8159761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14550 2022-11-23T03:35:22.8160614Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14551 2022-11-23T03:35:22.8161455Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14552 2022-11-23T03:35:22.8162644Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8163492Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8164612Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8165487Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8166307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:35:22.8167509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8168337Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8169468Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8170349Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8171656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:35:22.8172897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8173896Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8175043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8175936Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8176764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:35:22.8178020Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8178864Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8180004Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8180894Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8181715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:35:22.8182552Z skip: Need at least 4 CUDA devices (4.345s) 2022-11-23T03:35:22.8183423Z test_sharded_split (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14817 2022-11-23T03:35:22.8184397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14818 2022-11-23T03:35:22.8185227Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14819 2022-11-23T03:35:22.8186071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14820 2022-11-23T03:35:22.8187272Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8188108Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8189228Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8190112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8190923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:35:22.8192125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8192946Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8194078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8194950Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8195769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:35:22.8196982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8197820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8198961Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8199818Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8200634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:35:22.8201839Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8202663Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8203791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8204656Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8205463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:35:22.8206292Z skip: Need at least 4 CUDA devices (4.244s) 2022-11-23T03:35:22.8207189Z test_sharded_transpose (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15085 2022-11-23T03:35:22.8208167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15086 2022-11-23T03:35:22.8209004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15087 2022-11-23T03:35:22.8209843Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15088 2022-11-23T03:35:22.8211145Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8211994Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8213134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8214026Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8214978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:35:22.8216199Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8217041Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8218178Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8219048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8219874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:35:22.8221074Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8221897Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8223037Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8223905Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8224728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:35:22.8225944Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8226781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8227901Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8228768Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8229585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:35:22.8230316Z skip: Need at least 4 CUDA devices (4.147s) 2022-11-23T03:35:22.8231194Z test_sharded_view (__main__.TPShardingOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15353 2022-11-23T03:35:22.8232169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15354 2022-11-23T03:35:22.8233008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15355 2022-11-23T03:35:22.8233846Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15356 2022-11-23T03:35:22.8235043Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8235880Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8237015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8237897Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8238828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:35:22.8240055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8240894Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8242022Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8242891Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8243707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:35:22.8244913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8245741Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8246874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8247848Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8248668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:35:22.8249874Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:35:22.8250945Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:35:22.8252105Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:35:22.8252976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:35:22.8253814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:35:22.8254548Z skip: Need at least 4 CUDA devices (4.246s) 2022-11-23T03:35:22.8254902Z 2022-11-23T03:35:22.8255426Z ---------------------------------------------------------------------- 2022-11-23T03:35:22.8256038Z Ran 6 tests in 26.671s 2022-11-23T03:35:22.8256322Z 2022-11-23T03:35:22.8256500Z OK (skipped=6) 2022-11-23T03:35:22.8256775Z 2022-11-23T03:35:22.8256990Z Generating XML reports... 2022-11-23T03:35:22.8258211Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tp_sharding_ops/TEST-TPShardingOpsTest-20221123033453.xml 2022-11-23T03:35:22.8258880Z 2022-11-23T03:35:22.8259436Z ##[endgroup] 2022-11-23T03:35:22.8260663Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tp_sharding_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_tp_sharding_ops_iw8hjxwj) 2022-11-23T03:35:22.8261357Z 2022-11-23T03:35:22.8261890Z Running distributed/_tensor/test_redistribute ... [2022-11-23 03:35:22.808431] 2022-11-23T03:35:22.8263276Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:35:22.809120] 2022-11-23T03:36:00.5143463Z 2022-11-23T03:36:00.5143908Z Expand the folded group to see the log file of distributed/_tensor/test_redistribute 2022-11-23T03:36:00.5150513Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_redistribute_cn1kwusl) 2022-11-23T03:36:00.5153819Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_redistribute 2022-11-23T03:36:00.5154993Z 2022-11-23T03:36:00.5155396Z Running tests... 2022-11-23T03:36:00.5157356Z ---------------------------------------------------------------------- 2022-11-23T03:36:00.5159776Z test_multi_dim_mesh (__main__.MultiDimRedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15688 2022-11-23T03:36:00.5163218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15689 2022-11-23T03:36:00.5165658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15690 2022-11-23T03:36:00.5168092Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15691 2022-11-23T03:36:00.5170383Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 15692 2022-11-23T03:36:00.5172894Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 15693 2022-11-23T03:36:00.5175179Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 15694 2022-11-23T03:36:00.5177179Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 15695 2022-11-23T03:36:00.5180147Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5182148Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5184870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5186958Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5189331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:36:00.5192327Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5194544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5197154Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5199602Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5201365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:36:00.5204031Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5206291Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5209060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5211277Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5213268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:00.5216127Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5218207Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5220895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5221972Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5222993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:00.5225104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5225790Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5226762Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5227671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5228358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T03:36:00.5229310Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5229985Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5230846Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5231600Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5232525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T03:36:00.5233251Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5233691Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5234266Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5234724Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5235145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T03:36:00.5235767Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5236198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5236777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5237319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5237732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T03:36:00.5238112Z skip: Need at least 8 CUDA devices (5.278s) 2022-11-23T03:36:00.5238592Z test_partial_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16224 2022-11-23T03:36:00.5239123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16225 2022-11-23T03:36:00.5239742Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5240172Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5240747Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5241204Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5241615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:00.5242083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:00.5242708Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5243145Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5243719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5244174Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5244595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:00.5245057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:00.5245716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5246408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5246971Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5247465Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5247773Z ok (4.531s) 2022-11-23T03:36:00.5248180Z test_partial_to_shard (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16375 2022-11-23T03:36:00.5248675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16376 2022-11-23T03:36:00.5249287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5249785Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5250367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5251031Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5251509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:00.5252060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:00.5252815Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5253317Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5254003Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5254643Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5255150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:00.5255713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:00.5256506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5257332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5258002Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5258595Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5258964Z ok (4.732s) 2022-11-23T03:36:00.5259461Z test_replicate_to_partial (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16518 2022-11-23T03:36:00.5260089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16519 2022-11-23T03:36:00.5260825Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5261256Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5261826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5262276Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5262702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:00.5263170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:00.5263792Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5264232Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5264807Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5265259Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5265671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:00.5266134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:00.5266783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5267467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5268026Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5268517Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5269029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-11-23T03:36:00.5269495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-11-23T03:36:00.5270146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:36:00.5270659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-11-23T03:36:00.5271308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-11-23T03:36:00.5271815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-11-23T03:36:00.5272455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:36:00.5273021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-11-23T03:36:00.5273675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-11-23T03:36:00.5274170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-11-23T03:36:00.5274813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:36:00.5275491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-11-23T03:36:00.5275859Z ok (4.632s) 2022-11-23T03:36:00.5276298Z test_replicate_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16684 2022-11-23T03:36:00.5276834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16685 2022-11-23T03:36:00.5277456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5277890Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5278458Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5278912Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5279335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:00.5279803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:00.5280427Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5280858Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5281445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5281890Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5282312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:00.5282778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:00.5283430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5284110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5284670Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5285165Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5285466Z ok (4.631s) 2022-11-23T03:36:00.5285949Z test_replicate_to_shard_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16835 2022-11-23T03:36:00.5286480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16836 2022-11-23T03:36:00.5287096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5287526Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5288103Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5288558Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5288982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:00.5289440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:00.5290120Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5290624Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5291218Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5291766Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5292263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:00.5292813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:00.5293573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5294391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5295072Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5295665Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5296034Z ok (4.635s) 2022-11-23T03:36:00.5296554Z test_shard_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16986 2022-11-23T03:36:00.5297188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16987 2022-11-23T03:36:00.5297913Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5298434Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5299124Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5299668Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5300182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:00.5300749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:00.5301407Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:00.5301838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:00.5302405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:00.5302862Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:00.5303283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:00.5303750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:00.5304470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5305157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:00.5305718Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5306205Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:00.5306514Z ok (5.235s) 2022-11-23T03:36:00.5306647Z 2022-11-23T03:36:00.5306917Z ---------------------------------------------------------------------- 2022-11-23T03:36:00.5307231Z Ran 7 tests in 33.676s 2022-11-23T03:36:00.5307379Z 2022-11-23T03:36:00.5307470Z OK (skipped=1) 2022-11-23T03:36:00.5307606Z 2022-11-23T03:36:00.5307717Z Generating XML reports... 2022-11-23T03:36:00.5308314Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20221123033524.xml 2022-11-23T03:36:00.5309171Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20221123033524.xml 2022-11-23T03:36:00.5309525Z 2022-11-23T03:36:00.5309882Z ##[endgroup] 2022-11-23T03:36:00.5310498Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_redistribute_cn1kwusl) 2022-11-23T03:36:00.5310843Z 2022-11-23T03:36:00.5311114Z Running distributed/_tensor/test_matrix_ops ... [2022-11-23 03:36:00.514826] 2022-11-23T03:36:00.5311802Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:36:00.515501] 2022-11-23T03:36:51.9236797Z 2022-11-23T03:36:51.9238164Z Expand the folded group to see the log file of distributed/_tensor/test_matrix_ops 2022-11-23T03:36:51.9241462Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_matrix_ops_ubm7vrsw) 2022-11-23T03:36:51.9243843Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_matrix_ops 2022-11-23T03:36:51.9244577Z 2022-11-23T03:36:51.9244841Z Running tests... 2022-11-23T03:36:51.9245966Z ---------------------------------------------------------------------- 2022-11-23T03:36:51.9247280Z test_addmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17204 2022-11-23T03:36:51.9248730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17205 2022-11-23T03:36:51.9250464Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9251872Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9253509Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9254801Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9255973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:51.9257267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:51.9259012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9260192Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9261790Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9263014Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9264171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:51.9266125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:51.9268004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9269885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9271431Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9272786Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9273607Z ok (7.125s) 2022-11-23T03:36:51.9274729Z test_addmm_auto_redistribute (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17349 2022-11-23T03:36:51.9276121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17350 2022-11-23T03:36:51.9277764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9279198Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9280779Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9281998Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9283133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:51.9284386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:51.9286106Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9287281Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9288854Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9290099Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9291342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:51.9292632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:51.9294590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9296491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9298035Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9299388Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9300212Z ok (6.533s) 2022-11-23T03:36:51.9301264Z test_baddbmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17502 2022-11-23T03:36:51.9302622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17503 2022-11-23T03:36:51.9304292Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9305460Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9307055Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9308272Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9309416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:51.9310667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:51.9312379Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9313709Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9315298Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9316522Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9317662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:51.9318922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:51.9320700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9322547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9324082Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9325675Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9326494Z ok (8.444s) 2022-11-23T03:36:51.9327529Z test_bmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17655 2022-11-23T03:36:51.9328848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17656 2022-11-23T03:36:51.9330828Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9331996Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9333598Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9334830Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9335976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:51.9337259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:51.9338996Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9340157Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9341718Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9342942Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9344084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:51.9345339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:51.9346730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9347497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9348061Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9348560Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9348858Z ok (6.838s) 2022-11-23T03:36:51.9349242Z test_mm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17808 2022-11-23T03:36:51.9349731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17809 2022-11-23T03:36:51.9350343Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9350776Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9351354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9351873Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9352290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:51.9352760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:51.9353390Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9353823Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9354405Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9354860Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9355279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:51.9355738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:51.9356446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9357131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9357690Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9358187Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9358493Z ok (6.732s) 2022-11-23T03:36:51.9358876Z test_t (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17957 2022-11-23T03:36:51.9359351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17958 2022-11-23T03:36:51.9359962Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9360400Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9360977Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9361427Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9361845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:51.9362308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:51.9362927Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9363361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9363939Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9364410Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9364856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:51.9365322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:51.9365972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9366660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9367215Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9367711Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9368015Z ok (4.730s) 2022-11-23T03:36:51.9368412Z test_t_partial (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18100 2022-11-23T03:36:51.9368963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18101 2022-11-23T03:36:51.9369588Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9370020Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9370638Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9371094Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9371578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:36:51.9372133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:36:51.9372892Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:36:51.9373414Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:36:51.9374181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:36:51.9374719Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:36:51.9375219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:36:51.9375775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:36:51.9376560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9377383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:36:51.9378061Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9378664Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:36:51.9379037Z ok (7.039s) 2022-11-23T03:36:51.9379190Z 2022-11-23T03:36:51.9379515Z ---------------------------------------------------------------------- 2022-11-23T03:36:51.9379894Z Ran 7 tests in 47.442s 2022-11-23T03:36:51.9380069Z 2022-11-23T03:36:51.9380162Z OK 2022-11-23T03:36:51.9380300Z 2022-11-23T03:36:51.9380432Z Generating XML reports... 2022-11-23T03:36:51.9381147Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_matrix_ops/TEST-DistMatrixOpsTest-20221123033602.xml 2022-11-23T03:36:51.9381542Z 2022-11-23T03:36:51.9381879Z ##[endgroup] 2022-11-23T03:36:51.9382599Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_matrix_ops_ubm7vrsw) 2022-11-23T03:36:51.9383000Z 2022-11-23T03:36:51.9383316Z Running distributed/_tensor/test_dtensor ... [2022-11-23 03:36:51.924175] 2022-11-23T03:36:51.9384147Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:36:51.924904] 2022-11-23T03:37:52.8305701Z 2022-11-23T03:37:52.8306639Z Expand the folded group to see the log file of distributed/_tensor/test_dtensor 2022-11-23T03:37:52.8312380Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_dtensor_n_ecmhq7) 2022-11-23T03:37:52.8314593Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_dtensor 2022-11-23T03:37:52.8315312Z 2022-11-23T03:37:52.8315567Z Running tests... 2022-11-23T03:37:52.8316966Z ---------------------------------------------------------------------- 2022-11-23T03:37:52.8318952Z test_device_mesh_nd (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18312 2022-11-23T03:37:52.8321027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18313 2022-11-23T03:37:52.8323361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18314 2022-11-23T03:37:52.8324882Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18315 2022-11-23T03:37:52.8326248Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 18316 2022-11-23T03:37:52.8327505Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 18317 2022-11-23T03:37:52.8328783Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 18318 2022-11-23T03:37:52.8330241Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 18319 2022-11-23T03:37:52.8332618Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8334118Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8336099Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8338034Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8339418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8341528Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8342842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8344416Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8345696Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8347164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8349456Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8350950Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8352919Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8354447Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8355848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:37:52.8357855Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8359334Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8361024Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8362335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8363553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T03:37:52.8365358Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8366596Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8368254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8369528Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8370853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T03:37:52.8372519Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8373656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8375198Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8376385Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8377764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:37:52.8379445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8380584Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8382115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8383306Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8384417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T03:37:52.8386064Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8387219Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8388929Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8390120Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8391235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T03:37:52.8392234Z skip: Need at least 8 CUDA devices (5.333s) 2022-11-23T03:37:52.8393409Z test_dtensor_2d_mesh (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18848 2022-11-23T03:37:52.8394735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18849 2022-11-23T03:37:52.8395897Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18850 2022-11-23T03:37:52.8397056Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18851 2022-11-23T03:37:52.8398190Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 18852 2022-11-23T03:37:52.8399360Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 18853 2022-11-23T03:37:52.8400514Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 18854 2022-11-23T03:37:52.8401663Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 18855 2022-11-23T03:37:52.8403296Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8404439Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8405970Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8407149Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8408255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T03:37:52.8409895Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8411628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8413185Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8414380Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8415489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T03:37:52.8417104Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8418257Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8419786Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8420976Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8422288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:37:52.8423356Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8423842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8424417Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8424861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8425281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8425893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8443164Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8444060Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8444945Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8445463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8446328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8447033Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8447979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8448707Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8449418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T03:37:52.8450600Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8451381Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8452422Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8453358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8454212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T03:37:52.8455381Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8456217Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8457380Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8458336Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8459106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:37:52.8459855Z skip: Need at least 8 CUDA devices (4.583s) 2022-11-23T03:37:52.8460772Z test_dtensor_api_device_mesh_context_manager (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19384 2022-11-23T03:37:52.8461882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19385 2022-11-23T03:37:52.8462694Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19386 2022-11-23T03:37:52.8463484Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19387 2022-11-23T03:37:52.8464223Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 19388 2022-11-23T03:37:52.8464907Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 19389 2022-11-23T03:37:52.8465349Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 19390 2022-11-23T03:37:52.8465923Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 19391 2022-11-23T03:37:52.8466669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8467096Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8467676Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8468128Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8468549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:37:52.8469165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8469598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8470176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8470702Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8471114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T03:37:52.8471734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8472163Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8472738Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8473190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8473610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:37:52.8474225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8474656Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8475230Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8475681Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8476098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T03:37:52.8476713Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8477142Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8477719Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8478160Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8478581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8479202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8479630Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8480202Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8480650Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8481068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8481679Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8482104Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8482672Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8483182Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8483604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T03:37:52.8484227Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8484659Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8485234Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8485677Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8486093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T03:37:52.8486467Z skip: Need at least 8 CUDA devices (4.684s) 2022-11-23T03:37:52.8486949Z test_dtensor_device_mesh_device_conversion (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19920 2022-11-23T03:37:52.8487534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19921 2022-11-23T03:37:52.8487971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19922 2022-11-23T03:37:52.8488407Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19923 2022-11-23T03:37:52.8488835Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 19924 2022-11-23T03:37:52.8489269Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 19925 2022-11-23T03:37:52.8489704Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 19926 2022-11-23T03:37:52.8490138Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 19927 2022-11-23T03:37:52.8490858Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8491294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8491954Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8492490Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8493000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T03:37:52.8493740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8494260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8494952Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8495492Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8495998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:37:52.8496731Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8497247Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8497936Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8498479Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8498992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8499734Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8500251Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8500945Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8501478Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8502068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T03:37:52.8502817Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8503335Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8503910Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8504359Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8504778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T03:37:52.8505382Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8505812Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8506448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8506900Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8507318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:37:52.8507930Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8508363Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8508928Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8509376Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8509795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8510410Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8510842Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8511414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8511864Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8512285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T03:37:52.8512654Z skip: Need at least 8 CUDA devices (4.684s) 2022-11-23T03:37:52.8513123Z test_dtensor_spec_local_shard_offset (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20456 2022-11-23T03:37:52.8513641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20457 2022-11-23T03:37:52.8514080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20458 2022-11-23T03:37:52.8514521Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20459 2022-11-23T03:37:52.8514953Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 20460 2022-11-23T03:37:52.8515390Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 20461 2022-11-23T03:37:52.8515815Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 20462 2022-11-23T03:37:52.8516244Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 20463 2022-11-23T03:37:52.8516853Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8517284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8517860Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8518310Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8518791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8519403Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8519831Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8520406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8520855Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8521271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8521884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8522313Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8522885Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8523387Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8523805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5 2022-11-23T03:37:52.8524420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8524849Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8525424Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8525877Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8526288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:37:52.8526903Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8527338Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8527912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8528358Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8528776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6 2022-11-23T03:37:52.8529389Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8529820Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8530386Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8531327Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8531808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:37:52.8532558Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8533077Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8533768Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8534309Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8534801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4 2022-11-23T03:37:52.8535535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8536050Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8536739Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8537404Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8537910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7 2022-11-23T03:37:52.8538363Z skip: Need at least 8 CUDA devices (4.584s) 2022-11-23T03:37:52.8538889Z test_dtensor_constructor (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20992 2022-11-23T03:37:52.8539488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20993 2022-11-23T03:37:52.8540229Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8540744Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8541430Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8542038Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8542541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8543104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:37:52.8543735Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8544165Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8544740Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8545190Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8545607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8546075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:37:52.8546734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8547416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8547977Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8548475Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8548781Z ok (4.426s) 2022-11-23T03:37:52.8549181Z test_dtensor_properties (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21135 2022-11-23T03:37:52.8549679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21136 2022-11-23T03:37:52.8550287Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8550717Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8551290Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8551740Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8552156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8552620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:37:52.8553244Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8553671Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8554238Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8554687Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8555252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8555724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:37:52.8556379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8557067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8557628Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8558124Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8558420Z ok (4.530s) 2022-11-23T03:37:52.8558835Z test_dtensor_spec_read_only_after_set (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21278 2022-11-23T03:37:52.8559396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21279 2022-11-23T03:37:52.8560007Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8560438Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8561015Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8561467Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8561882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8562347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:37:52.8562971Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8563402Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8563981Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8564429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8564845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8565302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:37:52.8565955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8566638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8567201Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8567695Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8568009Z ok (4.931s) 2022-11-23T03:37:52.8568403Z test_dtensor_stride (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21421 2022-11-23T03:37:52.8568895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21422 2022-11-23T03:37:52.8569495Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8569924Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8570500Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8571021Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8571441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8571988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:37:52.8572822Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8573329Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8574029Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8574575Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8575082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8575639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:37:52.8576420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8577245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8577983Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8578581Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8578944Z ok (4.432s) 2022-11-23T03:37:52.8579412Z test_from_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21564 2022-11-23T03:37:52.8579994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21565 2022-11-23T03:37:52.8580728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8581242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8581926Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8582472Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8582985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8583510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:37:52.8584134Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8584563Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8585137Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8585584Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8585994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8586459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:37:52.8587113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8587794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8588354Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8588848Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8589151Z ok (5.036s) 2022-11-23T03:37:52.8589542Z test_from_local_then_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21711 2022-11-23T03:37:52.8590042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21712 2022-11-23T03:37:52.8590648Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8591083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8591715Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8592170Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8592588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8593045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:37:52.8593669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8594099Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8594675Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8595126Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8595598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8596062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:37:52.8596714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8597386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8597945Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8598436Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8598738Z ok (5.031s) 2022-11-23T03:37:52.8599121Z test_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21862 2022-11-23T03:37:52.8599603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21863 2022-11-23T03:37:52.8600219Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8600643Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8601221Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8601671Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8602091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:37:52.8602554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:37:52.8603176Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:37:52.8603604Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:37:52.8604179Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:37:52.8604630Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:37:52.8605046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:37:52.8605510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:37:52.8606161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8606841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:37:52.8607405Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8607897Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:37:52.8608199Z ok (4.631s) 2022-11-23T03:37:52.8608332Z 2022-11-23T03:37:52.8608660Z ---------------------------------------------------------------------- 2022-11-23T03:37:52.8608981Z Ran 12 tests in 56.887s 2022-11-23T03:37:52.8609131Z 2022-11-23T03:37:52.8609223Z OK (skipped=5) 2022-11-23T03:37:52.8609367Z 2022-11-23T03:37:52.8609475Z Generating XML reports... 2022-11-23T03:37:52.8610045Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20221123033653.xml 2022-11-23T03:37:52.8610919Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20221123033653.xml 2022-11-23T03:37:52.8611243Z 2022-11-23T03:37:52.8611653Z ##[endgroup] 2022-11-23T03:37:52.8612373Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_dtensor_n_ecmhq7) 2022-11-23T03:37:52.8612762Z 2022-11-23T03:37:52.8613085Z Running distributed/_tensor/test_common_rules ... [2022-11-23 03:37:52.831723] 2022-11-23T03:37:52.8614044Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:37:52.832398] 2022-11-23T03:38:45.7994671Z 2022-11-23T03:38:45.7995817Z Expand the folded group to see the log file of distributed/_tensor/test_common_rules 2022-11-23T03:38:45.7998015Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_common_rules_p0iyjz1g) 2022-11-23T03:38:45.8000534Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_common_rules 2022-11-23T03:38:45.8001272Z 2022-11-23T03:38:45.8003356Z Running tests... 2022-11-23T03:38:45.8005108Z ---------------------------------------------------------------------- 2022-11-23T03:38:45.8006502Z test_einop_basic_propagation (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22080 2022-11-23T03:38:45.8007934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22081 2022-11-23T03:38:45.8009125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22082 2022-11-23T03:38:45.8010331Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22083 2022-11-23T03:38:45.8012203Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8013385Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8014955Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8016179Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8017361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8019420Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8020781Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8022348Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8023572Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8024711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8027032Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8029243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8031367Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8032878Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8034715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8036448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8037589Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8039129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8040341Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8041457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8042444Z skip: Need at least 4 CUDA devices (5.122s) 2022-11-23T03:38:45.8043625Z test_einop_errors (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22348 2022-11-23T03:38:45.8044939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22349 2022-11-23T03:38:45.8046327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22350 2022-11-23T03:38:45.8047488Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22351 2022-11-23T03:38:45.8049142Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8050300Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8052081Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8053303Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8054422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8056075Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8057250Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8058791Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8060002Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8061107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8062758Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8063906Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8065449Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8066649Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8067773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8069451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8070598Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8072125Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8073319Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8074441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8075445Z skip: Need at least 4 CUDA devices (4.746s) 2022-11-23T03:38:45.8076630Z test_einop_linearity (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22616 2022-11-23T03:38:45.8077961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22617 2022-11-23T03:38:45.8079320Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22618 2022-11-23T03:38:45.8080473Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22619 2022-11-23T03:38:45.8082113Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8083260Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8084803Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8086010Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8087129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8088785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8089911Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8092012Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8093221Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8094353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8096014Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8097159Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8098275Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8098842Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8099346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8100096Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8100619Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8101314Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8101861Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8102367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8102822Z skip: Need at least 4 CUDA devices (4.346s) 2022-11-23T03:38:45.8103355Z test_einop_merge_sharding (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22884 2022-11-23T03:38:45.8103969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22885 2022-11-23T03:38:45.8104499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22886 2022-11-23T03:38:45.8105028Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22887 2022-11-23T03:38:45.8105764Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8106284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8106982Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8107519Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8108039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8108669Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8109103Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8109756Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8110216Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8110638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8111254Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8111685Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8112274Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8112727Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8113151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8113777Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8114268Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8114845Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8115296Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8115718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8116100Z skip: Need at least 4 CUDA devices (4.144s) 2022-11-23T03:38:45.8116564Z test_einop_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23152 2022-11-23T03:38:45.8117082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23153 2022-11-23T03:38:45.8117522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23154 2022-11-23T03:38:45.8117960Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23155 2022-11-23T03:38:45.8118575Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8119009Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8119589Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8120043Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8120464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8121084Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8121515Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8122088Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8122553Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8122975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8123595Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8124025Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8124602Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8125054Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8125466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8126089Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8126521Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8127155Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8127614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8128035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8128411Z skip: Need at least 4 CUDA devices (4.152s) 2022-11-23T03:38:45.8128871Z test_einop_pointwise_propagation (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23420 2022-11-23T03:38:45.8129398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23421 2022-11-23T03:38:45.8129833Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23422 2022-11-23T03:38:45.8130271Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23423 2022-11-23T03:38:45.8131328Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8131835Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8132526Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8133063Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8133555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8134294Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8134815Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8135513Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8136055Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8136573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8137325Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8137839Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8138535Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8139073Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8139575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8140317Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8140840Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8141540Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8142079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8142592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8143052Z skip: Need at least 4 CUDA devices (4.348s) 2022-11-23T03:38:45.8143659Z test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23688 2022-11-23T03:38:45.8144326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23689 2022-11-23T03:38:45.8144853Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23690 2022-11-23T03:38:45.8145382Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23691 2022-11-23T03:38:45.8146200Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8146728Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8147432Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8147984Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8148442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8149057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8149493Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8150071Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8150518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8151002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8151620Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8152051Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8152627Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8153081Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8153503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8154115Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8154544Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8155122Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8155582Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8156004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8156381Z skip: Need at least 4 CUDA devices (4.152s) 2022-11-23T03:38:45.8156853Z test_pointwise_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23956 2022-11-23T03:38:45.8157380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23957 2022-11-23T03:38:45.8157809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23958 2022-11-23T03:38:45.8158250Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23959 2022-11-23T03:38:45.8158863Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8159307Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8159884Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8160335Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8160752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8161361Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8161793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8162372Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8162824Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8163244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8163921Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8164361Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8164934Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8165393Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8165813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8166433Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8166870Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8167448Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8167960Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8168375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8168877Z skip: Need at least 4 CUDA devices (4.248s) 2022-11-23T03:38:45.8169341Z test_pointwise_rules_broadcasting (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24224 2022-11-23T03:38:45.8169865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24225 2022-11-23T03:38:45.8170306Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24226 2022-11-23T03:38:45.8170832Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24227 2022-11-23T03:38:45.8171469Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8171998Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8172696Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8173236Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8173741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8174493Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8175024Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8175728Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8176280Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8176798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8177569Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8178083Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8178785Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8179339Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8179854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8180608Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8181135Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8181841Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8182382Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8183014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8183484Z skip: Need at least 4 CUDA devices (4.344s) 2022-11-23T03:38:45.8184043Z test_pointwise_rules_suggestion (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24492 2022-11-23T03:38:45.8184657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24493 2022-11-23T03:38:45.8185183Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24494 2022-11-23T03:38:45.8185708Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24495 2022-11-23T03:38:45.8186445Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8186966Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8187743Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8188295Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8188752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8189371Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8189801Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8190377Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8190819Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8191244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8191865Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8192294Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8192870Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8193325Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8193745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8194354Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8194788Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8195366Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8195815Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8196245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8196623Z skip: Need at least 4 CUDA devices (4.246s) 2022-11-23T03:38:45.8197071Z test_reduction_rule (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24760 2022-11-23T03:38:45.8197560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24761 2022-11-23T03:38:45.8198000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24762 2022-11-23T03:38:45.8198436Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24763 2022-11-23T03:38:45.8199045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8199480Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8200057Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8200574Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8200989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:45.8201614Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8202046Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8202623Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8203079Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8203502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:38:45.8204119Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8204611Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8205182Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8205632Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8206055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:45.8206674Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:45.8207105Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:45.8207682Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:45.8208135Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:45.8208546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:38:45.8208932Z skip: Need at least 4 CUDA devices (5.052s) 2022-11-23T03:38:45.8209113Z 2022-11-23T03:38:45.8209386Z ---------------------------------------------------------------------- 2022-11-23T03:38:45.8209704Z Ran 11 tests in 48.904s 2022-11-23T03:38:45.8209854Z 2022-11-23T03:38:45.8209948Z OK (skipped=11) 2022-11-23T03:38:45.8210095Z 2022-11-23T03:38:45.8210207Z Generating XML reports... 2022-11-23T03:38:45.8210940Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20221123033754.xml 2022-11-23T03:38:45.8211291Z 2022-11-23T03:38:45.8211671Z ##[endgroup] 2022-11-23T03:38:45.8212409Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-test_common_rules_p0iyjz1g) 2022-11-23T03:38:45.8212814Z 2022-11-23T03:38:45.8213200Z Running distributed/_tensor/parallel/test_view_sharding_dim_change ... [2022-11-23 03:38:45.800291] 2022-11-23T03:38:45.8214144Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_view_sharding_dim_change.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:38:45.800968] 2022-11-23T03:38:54.9193729Z 2022-11-23T03:38:54.9195337Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_view_sharding_dim_change 2022-11-23T03:38:54.9198246Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_view_sharding_dim_change_vzsqu9ru) 2022-11-23T03:38:54.9201468Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_view_sharding_dim_change 2022-11-23T03:38:54.9202467Z 2022-11-23T03:38:54.9202726Z Running tests... 2022-11-23T03:38:54.9203887Z ---------------------------------------------------------------------- 2022-11-23T03:38:54.9205925Z test_view_with_sharding_dim_change (__main__.TPViewShardingDimChangeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25095 2022-11-23T03:38:54.9207681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25096 2022-11-23T03:38:54.9209460Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:54.9211356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:54.9212979Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:54.9214208Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:54.9215358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:38:54.9216613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:38:54.9218351Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:38:54.9219811Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:38:54.9221406Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:38:54.9222628Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:38:54.9223763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:38:54.9225031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:38:54.9226841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:54.9228750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:38:54.9230718Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:54.9232393Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:38:54.9233241Z ok (5.115s) 2022-11-23T03:38:54.9233597Z 2022-11-23T03:38:54.9234307Z ---------------------------------------------------------------------- 2022-11-23T03:38:54.9235145Z Ran 1 test in 5.115s 2022-11-23T03:38:54.9235536Z 2022-11-23T03:38:54.9235748Z OK 2022-11-23T03:38:54.9236065Z 2022-11-23T03:38:54.9236356Z Generating XML reports... 2022-11-23T03:38:54.9238275Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20221123033847.xml 2022-11-23T03:38:54.9239342Z 2022-11-23T03:38:54.9240085Z ##[endgroup] 2022-11-23T03:38:54.9242057Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_view_sharding_dim_change_vzsqu9ru) 2022-11-23T03:38:54.9243167Z 2022-11-23T03:38:54.9243986Z Running distributed/_tensor/parallel/test_tp_examples ... [2022-11-23 03:38:54.919603] 2022-11-23T03:38:54.9246023Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_tp_examples.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:38:54.920313] 2022-11-23T03:39:20.5033487Z 2022-11-23T03:39:20.5034548Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_tp_examples 2022-11-23T03:39:20.5036882Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_examples (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_tp_examples_lzqy8b6o) 2022-11-23T03:39:20.5040683Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_tp_examples 2022-11-23T03:39:20.5041734Z 2022-11-23T03:39:20.5041997Z Running tests... 2022-11-23T03:39:20.5043559Z ---------------------------------------------------------------------- 2022-11-23T03:39:20.5045878Z test_mlp_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25309 2022-11-23T03:39:20.5047615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25310 2022-11-23T03:39:20.5050045Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:20.5051407Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:20.5053165Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:20.5054431Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:20.5055902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:20.5057454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:20.5060284Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:20.5061793Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:20.5063722Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:20.5065033Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:20.5066251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:20.5067594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:20.5069505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:20.5071460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:20.5073045Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:20.5074374Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:20.5075176Z ok (7.218s) 2022-11-23T03:39:20.5076348Z test_self_attn_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25462 2022-11-23T03:39:20.5077881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25463 2022-11-23T03:39:20.5079517Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:20.5080678Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:20.5082226Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:20.5083429Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:20.5084569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:20.5085818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:20.5087481Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:20.5088637Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:20.5090181Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:20.5091497Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:20.5092632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:20.5093884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:20.5095856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:20.5097703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:20.5099213Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:20.5100520Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:20.5101337Z ok (7.438s) 2022-11-23T03:39:20.5102561Z test_self_attn_replacement_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25615 2022-11-23T03:39:20.5104046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25616 2022-11-23T03:39:20.5105680Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:20.5106821Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:20.5108530Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:20.5109744Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:20.5110867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:20.5112114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:20.5113780Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:20.5114932Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:20.5116477Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:20.5117665Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:20.5118805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:20.5120046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:20.5121784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:20.5123602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:20.5125095Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:20.5126407Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2022-11-23T03:39:20.5127202Z ok (6.637s) 2022-11-23T03:39:20.5127554Z 2022-11-23T03:39:20.5128265Z ---------------------------------------------------------------------- 2022-11-23T03:39:20.5129091Z Ran 3 tests in 21.293s 2022-11-23T03:39:20.5129497Z 2022-11-23T03:39:20.5129700Z OK 2022-11-23T03:39:20.5130012Z 2022-11-23T03:39:20.5130305Z Generating XML reports... 2022-11-23T03:39:20.5132436Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_tp_examples/TEST-DistTensorParallelExampleTest-20221123033856.xml 2022-11-23T03:39:20.5133454Z 2022-11-23T03:39:20.5134345Z ##[endgroup] 2022-11-23T03:39:20.5136092Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_examples (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_tp_examples_lzqy8b6o) 2022-11-23T03:39:20.5137101Z 2022-11-23T03:39:20.5137890Z Running distributed/_tensor/parallel/test_2d_parallel ... [2022-11-23 03:39:20.503740] 2022-11-23T03:39:20.5139840Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_2d_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:39:20.504432] 2022-11-23T03:39:33.7272850Z 2022-11-23T03:39:33.7274149Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_2d_parallel 2022-11-23T03:39:33.7277472Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_2d_parallel (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_2d_parallel_nsgshxcg) 2022-11-23T03:39:33.7279848Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel 2022-11-23T03:39:33.7280612Z 2022-11-23T03:39:33.7280859Z Running tests... 2022-11-23T03:39:33.7282402Z ---------------------------------------------------------------------- 2022-11-23T03:39:33.7284364Z test_2d_fsdp_integration_correctness (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25835 2022-11-23T03:39:33.7286202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25836 2022-11-23T03:39:33.7288059Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:33.7289552Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:33.7291299Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:33.7292518Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:33.7293651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:33.7294897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:33.7296701Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:33.7297859Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:33.7299414Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:33.7300614Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:33.7301756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:33.7303010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:33.7304766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:33.7306615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:33.7307736Z skip: Need at least 4 CUDA devices (5.106s) 2022-11-23T03:39:33.7309049Z test_2d_fsdp_integration_functionality (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25971 2022-11-23T03:39:33.7310501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25972 2022-11-23T03:39:33.7312129Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:33.7313284Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:33.7314834Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:33.7316048Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:33.7317172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:33.7318417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:39:33.7320102Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:33.7321238Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:33.7322793Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:33.7324192Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:33.7325322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:33.7326560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:39:33.7328326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:33.7330161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-11-23T03:39:33.7331449Z skip: Need at least 4 CUDA devices (4.030s) 2022-11-23T03:39:33.7331907Z 2022-11-23T03:39:33.7332627Z ---------------------------------------------------------------------- 2022-11-23T03:39:33.7333450Z Ran 2 tests in 9.137s 2022-11-23T03:39:33.7333840Z 2022-11-23T03:39:33.7334241Z OK (skipped=2) 2022-11-23T03:39:33.7334610Z 2022-11-23T03:39:33.7334901Z Generating XML reports... 2022-11-23T03:39:33.7336640Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20221123033922.xml 2022-11-23T03:39:33.7337624Z 2022-11-23T03:39:33.7338359Z ##[endgroup] 2022-11-23T03:39:33.7340092Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_2d_parallel (/var/lib/jenkins/pytorch/test/test-reports/distributed-_tensor-parallel-test_2d_parallel_nsgshxcg) 2022-11-23T03:39:33.7341082Z 2022-11-23T03:39:33.7341904Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-11-23 03:39:33.727640] 2022-11-23T03:39:33.7343909Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:39:33.728329] 2022-11-23T03:39:51.4791360Z 2022-11-23T03:39:51.4792484Z Expand the folded group to see the log file of distributed/_shard/sharding_spec/test_sharding_spec 2022-11-23T03:39:51.4795555Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_6cedfjtf) 2022-11-23T03:39:51.4798006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3r0nf21 2022-11-23T03:39:51.4799799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3r0nf21/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4802370Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec 2022-11-23T03:39:51.4803321Z 2022-11-23T03:39:51.4803646Z Running tests... 2022-11-23T03:39:51.4804965Z ---------------------------------------------------------------------- 2022-11-23T03:39:51.4806384Z test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26174 2022-11-23T03:39:51.4808033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26175 2022-11-23T03:39:51.4809458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26176 2022-11-23T03:39:51.4810921Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26177 2022-11-23T03:39:51.4812657Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4813838Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4815435Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4816657Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4817852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpan12fv_k 2022-11-23T03:39:51.4819230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpan12fv_k/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4820968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:51.4822733Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4823908Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4825503Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4826722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4827919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm94fh4ix 2022-11-23T03:39:51.4829280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm94fh4ix/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4830578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:39:51.4832514Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4833681Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4835263Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4836485Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4837673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpue5vobmw 2022-11-23T03:39:51.4839062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpue5vobmw/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4840361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:39:51.4842073Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4843242Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4844833Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4846052Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4847227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5odbr38a 2022-11-23T03:39:51.4848600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5odbr38a/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4849920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:51.4850888Z ok (4.642s) 2022-11-23T03:39:51.4851752Z test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec) 2022-11-23T03:39:51.4853065Z Test custom spec can be invoked from the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26442 2022-11-23T03:39:51.4854388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26443 2022-11-23T03:39:51.4855599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26444 2022-11-23T03:39:51.4856786Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26445 2022-11-23T03:39:51.4858443Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4859628Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4861196Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4862415Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4863593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1eg0zwe5 2022-11-23T03:39:51.4864966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1eg0zwe5/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4866451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:51.4868183Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4869348Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4870915Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4872130Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4873314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hrujir6 2022-11-23T03:39:51.4874675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hrujir6/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4875993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:51.4877705Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4879010Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4880611Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4881828Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4883021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxehx9ew 2022-11-23T03:39:51.4884397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxehx9ew/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4885730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:39:51.4887451Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4888615Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4890208Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4891722Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4892906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpax6e7voq 2022-11-23T03:39:51.4894272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpax6e7voq/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4895713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:39:51.4896737Z skip: Need at least 4 CUDA devices (4.150s) 2022-11-23T03:39:51.4897712Z test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec) 2022-11-23T03:39:51.4899050Z Test sharded_tensor.ones(...) with the custom ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26710 2022-11-23T03:39:51.4900360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26711 2022-11-23T03:39:51.4901555Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26712 2022-11-23T03:39:51.4902731Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26713 2022-11-23T03:39:51.4904340Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4904879Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4905489Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4905943Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4906376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w93lm3m 2022-11-23T03:39:51.4906885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w93lm3m/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4907443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:39:51.4908078Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4908510Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4909093Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4909549Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4909981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_nbwdyos 2022-11-23T03:39:51.4910493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_nbwdyos/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4910985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:39:51.4911610Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4912098Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4912687Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4913141Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4913584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3apex9w 2022-11-23T03:39:51.4914084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3apex9w/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4914737Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:39:51.4915171Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:39:51.4915754Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:39:51.4916218Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:39:51.4916666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcxklezi 2022-11-23T03:39:51.4917176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcxklezi/_remote_module_non_scriptable.py 2022-11-23T03:39:51.4917657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:39:51.4918114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:39:51.4918491Z skip: Need at least 4 CUDA devices (4.548s) 2022-11-23T03:39:51.4918841Z test_check_overlapping (__main__.TestShardingSpec) ... ok (0.008s) 2022-11-23T03:39:51.4919232Z test_chunked_sharding_spec (__main__.TestShardingSpec) ... ok (0.015s) 2022-11-23T03:39:51.4919621Z test_device_placement (__main__.TestShardingSpec) ... ok (0.006s) 2022-11-23T03:39:51.4920020Z test_enumerable_sharding_spec (__main__.TestShardingSpec) ... ok (0.036s) 2022-11-23T03:39:51.4920417Z test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.004s) 2022-11-23T03:39:51.4920809Z test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.002s) 2022-11-23T03:39:51.4921183Z test_get_split_size (__main__.TestShardingSpec) ... ok (0.002s) 2022-11-23T03:39:51.4921594Z test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.010s) 2022-11-23T03:39:51.4921841Z 2022-11-23T03:39:51.4922120Z ---------------------------------------------------------------------- 2022-11-23T03:39:51.4922434Z Ran 11 tests in 13.425s 2022-11-23T03:39:51.4922587Z 2022-11-23T03:39:51.4922680Z OK (skipped=2) 2022-11-23T03:39:51.4922813Z 2022-11-23T03:39:51.4922922Z Generating XML reports... 2022-11-23T03:39:51.4923577Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221123033935.xml 2022-11-23T03:39:51.4924465Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221123033935.xml 2022-11-23T03:39:51.4924814Z 2022-11-23T03:39:51.4925089Z ##[endgroup] 2022-11-23T03:39:51.4925774Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/pytorch/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_6cedfjtf) 2022-11-23T03:39:51.4926158Z 2022-11-23T03:39:51.4926441Z Running distributed/_composable/test_replicate ... [2022-11-23 03:39:51.479832] 2022-11-23T03:39:51.4927160Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:39:51.480469] 2022-11-23T03:40:05.7409005Z 2022-11-23T03:40:05.7410158Z Expand the folded group to see the log file of distributed/_composable/test_replicate 2022-11-23T03:40:05.7413465Z ##[group]PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_replicate_dr03exf0) 2022-11-23T03:40:05.7417008Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_replicate 2022-11-23T03:40:05.7417760Z 2022-11-23T03:40:05.7418010Z Running tests... 2022-11-23T03:40:05.7419118Z ---------------------------------------------------------------------- 2022-11-23T03:40:05.7420613Z test_replicate_multi_module (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27045 2022-11-23T03:40:05.7421997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27046 2022-11-23T03:40:05.7423186Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27047 2022-11-23T03:40:05.7424366Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27048 2022-11-23T03:40:05.7426050Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:05.7427239Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:05.7428801Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:05.7430025Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:05.7431162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:40:05.7432826Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:05.7433988Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:05.7435522Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:05.7436738Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:05.7437886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:40:05.7439557Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:05.7440705Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:05.7442256Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:05.7443466Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:05.7444574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:40:05.7446225Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:05.7447379Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:05.7449201Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:05.7450428Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:05.7451656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:05.7452905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:05.7454174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T03:40:05.7455408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:40:05.7456658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T03:40:05.7458443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:40:05.7460299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:40:05.7462316Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:40:05.7464140Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:40:05.7465480Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:40:05.7466727Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:40:05.7467935Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:40:05.7469177Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:40:05.7470055Z ok (5.669s) 2022-11-23T03:40:05.7471162Z test_replicate_single_module (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27457 2022-11-23T03:40:05.7472531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27458 2022-11-23T03:40:05.7473694Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27459 2022-11-23T03:40:05.7474855Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27460 2022-11-23T03:40:05.7476486Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:05.7477646Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:05.7479197Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:05.7480412Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:05.7481537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-11-23T03:40:05.7483216Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:05.7484368Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:05.7485893Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:05.7487108Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:05.7488237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-11-23T03:40:05.7489897Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:05.7491356Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:05.7492912Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:05.7494112Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:05.7495418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-11-23T03:40:05.7497094Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests 2022-11-23T03:40:05.7498243Z warnings.warn(f"loaded {len(slow_tests_dict)} slow tests") 2022-11-23T03:40:05.7499778Z /opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests 2022-11-23T03:40:05.7500983Z warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests") 2022-11-23T03:40:05.7502103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-11-23T03:40:05.7503342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-23T03:40:05.7504608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-23T03:40:05.7505716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-23T03:40:05.7506468Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:40:05.7506986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-11-23T03:40:05.7507640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:40:05.7508325Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:40:05.7509016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-11-23T03:40:05.7509517Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:40:05.7509989Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:40:05.7510444Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:40:05.7510915Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration. 2022-11-23T03:40:05.7511247Z ok (4.647s) 2022-11-23T03:40:05.7511382Z 2022-11-23T03:40:05.7511654Z ---------------------------------------------------------------------- 2022-11-23T03:40:05.7511966Z Ran 2 tests in 10.317s 2022-11-23T03:40:05.7512116Z 2022-11-23T03:40:05.7512193Z OK 2022-11-23T03:40:05.7512312Z 2022-11-23T03:40:05.7512425Z Generating XML reports... 2022-11-23T03:40:05.7513008Z Generated XML report: test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20221123033953.xml 2022-11-23T03:40:05.7513335Z 2022-11-23T03:40:05.7513609Z ##[endgroup] 2022-11-23T03:40:05.7514289Z FINISHED PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_replicate_dr03exf0) 2022-11-23T03:40:05.7514721Z 2022-11-23T03:40:05.7515078Z Running distributed/_composable/test_contract ... [2022-11-23 03:40:05.741044] 2022-11-23T03:40:05.7515848Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_contract.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 03:40:05.741793] 2022-11-23T03:40:10.3052500Z 2022-11-23T03:40:10.3054396Z Expand the folded group to see the log file of distributed/_composable/test_contract 2022-11-23T03:40:10.3057322Z ##[group]PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_contract_p4ls7t_n) 2022-11-23T03:40:10.3059637Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_contract 2022-11-23T03:40:10.3060423Z 2022-11-23T03:40:10.3060731Z Running tests... 2022-11-23T03:40:10.3061936Z ---------------------------------------------------------------------- 2022-11-23T03:40:10.3063580Z test_add_hooks (__main__.TestContract) ... ok (0.601s) 2022-11-23T03:40:10.3064518Z test_modify_fqn (__main__.TestContract) ... ok (0.002s) 2022-11-23T03:40:10.3065405Z test_state (__main__.TestContract) ... ok (0.002s) 2022-11-23T03:40:10.3065911Z 2022-11-23T03:40:10.3066662Z ---------------------------------------------------------------------- 2022-11-23T03:40:10.3067533Z Ran 3 tests in 0.606s 2022-11-23T03:40:10.3067920Z 2022-11-23T03:40:10.3068133Z OK 2022-11-23T03:40:10.3068451Z 2022-11-23T03:40:10.3068747Z Generating XML reports... 2022-11-23T03:40:10.3070375Z Generated XML report: test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20221123034007.xml 2022-11-23T03:40:10.3071263Z 2022-11-23T03:40:10.3072011Z ##[endgroup] 2022-11-23T03:40:10.3073731Z FINISHED PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/pytorch/test/test-reports/distributed-_composable-test_contract_p4ls7t_n) 2022-11-23T03:40:10.3074954Z 2022-11-23T03:40:12.5737993Z 2022-11-23T03:40:12.5738746Z real 124m0.177s 2022-11-23T03:40:12.5739394Z user 199m48.507s 2022-11-23T03:40:12.5740028Z sys 120m21.194s 2022-11-23T03:40:12.5740661Z + assert_git_not_dirty 2022-11-23T03:40:12.5741784Z + [[ linux-focal-rocm5.2-py3.8 != *rocm* ]] 2022-11-23T03:40:12.5742828Z + [[ linux-focal-rocm5.2-py3.8 == *cuda* ]] 2022-11-23T03:40:12.5743551Z + [[ 2 == 1 ]] 2022-11-23T03:40:12.5887675Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2022-11-23T03:40:12.5888706Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2022-11-23T03:40:12.5889796Z docker exec -t "8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2022-11-23T03:40:12.5926179Z shell: /bin/bash -e {0} 2022-11-23T03:40:12.5926636Z env: 2022-11-23T03:40:12.5927064Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:40:12.5927627Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:40:12.5928524Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:40:12.5929510Z CONTAINER_NAME: 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:40:12.5930130Z ##[endgroup] 2022-11-23T03:40:12.7175281Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2022-11-23T03:40:12.7682217Z Prepare all required actions 2022-11-23T03:40:12.7682966Z Getting action download info 2022-11-23T03:40:13.1905050Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2022-11-23T03:40:14.0597097Z ##[group]Run ./.github/actions/get-workflow-job-id 2022-11-23T03:40:14.0597616Z with: 2022-11-23T03:40:14.0598460Z github-token: *** 2022-11-23T03:40:14.0598872Z env: 2022-11-23T03:40:14.0599287Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:40:14.0599837Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:40:14.0600714Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:40:14.0601692Z CONTAINER_NAME: 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:40:14.0602307Z ##[endgroup] 2022-11-23T03:40:14.0645065Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2022-11-23T03:40:14.0645358Z with: 2022-11-23T03:40:14.0645563Z shell: bash 2022-11-23T03:40:14.0645792Z timeout_minutes: 10 2022-11-23T03:40:14.0646027Z max_attempts: 5 2022-11-23T03:40:14.0646265Z retry_wait_seconds: 30 2022-11-23T03:40:14.0646772Z command: set -eux python3 -m pip install requests==2.26.0 GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}" 2022-11-23T03:40:14.0647276Z polling_interval_seconds: 1 2022-11-23T03:40:14.0647530Z warning_on_retry: true 2022-11-23T03:40:14.0647779Z continue_on_error: false 2022-11-23T03:40:14.0648010Z env: 2022-11-23T03:40:14.0648230Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:40:14.0648523Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:40:14.0648979Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:40:14.0649503Z CONTAINER_NAME: 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:40:14.0650114Z GITHUB_TOKEN: *** 2022-11-23T03:40:14.0650346Z ##[endgroup] 2022-11-23T03:40:14.1358547Z + python3 -m pip install requests==2.26.0 2022-11-23T03:40:15.0371742Z Collecting requests==2.26.0 2022-11-23T03:40:15.1534809Z Using cached https://files.pythonhosted.org/packages/92/96/144f70b972a9c0eabbd4391ef93ccd49d0f2747f4f6a2a2738e99e5adc65/requests-2.26.0-py2.py3-none-any.whl 2022-11-23T03:40:15.1711474Z Collecting idna<4,>=2.5; python_version >= "3" (from requests==2.26.0) 2022-11-23T03:40:15.2043387Z Using cached https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl 2022-11-23T03:40:15.2084377Z Collecting urllib3<1.27,>=1.21.1 (from requests==2.26.0) 2022-11-23T03:40:15.2724520Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T03:40:15.2967365Z Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests==2.26.0) 2022-11-23T03:40:15.4289773Z Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl 2022-11-23T03:40:15.4339164Z Collecting certifi>=2017.4.17 (from requests==2.26.0) 2022-11-23T03:40:15.4798274Z Using cached https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl 2022-11-23T03:40:15.4843112Z Installing collected packages: idna, urllib3, charset-normalizer, certifi, requests 2022-11-23T03:40:15.7046500Z Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 idna-3.4 requests-2.27.1 urllib3-1.26.12 2022-11-23T03:40:15.7471240Z ++ python3 .github/scripts/get_workflow_job_id.py 3528394938 worker-rocm-amd-90 2022-11-23T03:40:18.0284867Z + GHA_WORKFLOW_JOB_ID=9655437853 2022-11-23T03:40:18.0286094Z + echo job-id=9655437853 2022-11-23T03:40:18.1343252Z Command completed after 1 attempt(s). 2022-11-23T03:40:18.1554256Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2022-11-23T03:40:18.1554575Z kill "$MONITOR_SCRIPT_PID" 2022-11-23T03:40:18.1576404Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T03:40:18.1576742Z env: 2022-11-23T03:40:18.1577014Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:40:18.1577376Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:40:18.1577950Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:40:18.1578581Z CONTAINER_NAME: 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:40:18.1579008Z MONITOR_SCRIPT_PID: 46644 2022-11-23T03:40:18.1579297Z ##[endgroup] 2022-11-23T03:40:18.1738070Z Prepare all required actions 2022-11-23T03:40:18.1738461Z Getting action download info 2022-11-23T03:40:18.4099529Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2022-11-23T03:40:19.4697369Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb) 2022-11-23T03:40:20.3354018Z ##[group]Run ./.github/actions/upload-test-artifacts 2022-11-23T03:40:20.3354297Z with: 2022-11-23T03:40:20.3354509Z use-gha: true 2022-11-23T03:40:20.3354840Z file-suffix: test-distributed-2-2-linux.rocm.gpu_9655437853 2022-11-23T03:40:20.3355139Z env: 2022-11-23T03:40:20.3355367Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:40:20.3355656Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:40:20.3356126Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:40:20.3356673Z CONTAINER_NAME: 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:40:20.3357010Z ##[endgroup] 2022-11-23T03:40:20.3424413Z ##[group]Run actions/upload-artifact@v3 2022-11-23T03:40:20.3424679Z with: 2022-11-23T03:40:20.3425041Z name: test-jsons-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437853.zip 2022-11-23T03:40:20.3425421Z retention-days: 14 2022-11-23T03:40:20.3425668Z if-no-files-found: warn 2022-11-23T03:40:20.3425920Z path: test/**/*.json 2022-11-23T03:40:20.3426140Z env: 2022-11-23T03:40:20.3426366Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:40:20.3426813Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:40:20.3427284Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:40:20.3427794Z CONTAINER_NAME: 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:40:20.3428125Z ##[endgroup] 2022-11-23T03:40:20.6356031Z With the provided path, there will be 3 files uploaded 2022-11-23T03:40:20.6358960Z Starting artifact upload 2022-11-23T03:40:20.6361228Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T03:40:20.6362749Z Artifact name is valid! 2022-11-23T03:40:20.7607838Z Container for artifact "test-jsons-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437853.zip" successfully created. Starting upload of file(s) 2022-11-23T03:40:21.1548010Z Total size of all the files uploaded is 29304 bytes 2022-11-23T03:40:21.1549153Z File upload process has finished. Finalizing the artifact upload 2022-11-23T03:40:21.4496097Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T03:40:21.4496788Z 2022-11-23T03:40:21.4497297Z The raw size of all the files that were specified for upload is 301646 bytes 2022-11-23T03:40:21.4498678Z The size of all the files that were uploaded is 29304 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T03:40:21.4499532Z 2022-11-23T03:40:21.4501068Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T03:40:21.4502103Z 2022-11-23T03:40:21.4503282Z Artifact test-jsons-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437853.zip has been successfully uploaded! 2022-11-23T03:40:21.4628448Z ##[group]Run actions/upload-artifact@v3 2022-11-23T03:40:21.4628954Z with: 2022-11-23T03:40:21.4629663Z name: test-reports-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437853.zip 2022-11-23T03:40:21.4630419Z retention-days: 14 2022-11-23T03:40:21.4630901Z if-no-files-found: ignore 2022-11-23T03:40:21.4631419Z path: test/**/*.xml test/**/*.csv 2022-11-23T03:40:21.4631878Z env: 2022-11-23T03:40:21.4632308Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:40:21.4633075Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:40:21.4633977Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:40:21.4634966Z CONTAINER_NAME: 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:40:21.4635596Z ##[endgroup] 2022-11-23T03:40:21.8141726Z With the provided path, there will be 803 files uploaded 2022-11-23T03:40:21.8142290Z Starting artifact upload 2022-11-23T03:40:21.8143089Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T03:40:21.8143668Z Artifact name is valid! 2022-11-23T03:40:21.9710556Z Container for artifact "test-reports-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437853.zip" successfully created. Starting upload of file(s) 2022-11-23T03:40:31.9884491Z Total file count: 803 ---- Processed file #105 (13.0%) 2022-11-23T03:40:41.9894121Z Total file count: 803 ---- Processed file #211 (26.2%) 2022-11-23T03:40:51.9903576Z Total file count: 803 ---- Processed file #323 (40.2%) 2022-11-23T03:41:01.9907599Z Total file count: 803 ---- Processed file #432 (53.7%) 2022-11-23T03:41:11.9911228Z Total file count: 803 ---- Processed file #543 (67.6%) 2022-11-23T03:41:21.9908041Z Total file count: 803 ---- Processed file #662 (82.4%) 2022-11-23T03:41:31.9914406Z Total file count: 803 ---- Processed file #778 (96.8%) 2022-11-23T03:41:34.2032437Z Total size of all the files uploaded is 254511 bytes 2022-11-23T03:41:34.2033479Z File upload process has finished. Finalizing the artifact upload 2022-11-23T03:41:34.2902386Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T03:41:34.2903047Z 2022-11-23T03:41:34.2903558Z The raw size of all the files that were specified for upload is 519206 bytes 2022-11-23T03:41:34.2904949Z The size of all the files that were uploaded is 254511 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T03:41:34.2905809Z 2022-11-23T03:41:34.2910451Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T03:41:34.2911552Z 2022-11-23T03:41:34.2912807Z Artifact test-reports-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437853.zip has been successfully uploaded! 2022-11-23T03:41:34.3073696Z ##[group]Run actions/upload-artifact@v3 2022-11-23T03:41:34.3074291Z with: 2022-11-23T03:41:34.3075101Z name: usage-log-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437853.zip 2022-11-23T03:41:34.3075962Z retention-days: 14 2022-11-23T03:41:34.3076545Z if-no-files-found: ignore 2022-11-23T03:41:34.3077125Z path: usage_log.txt 2022-11-23T03:41:34.3077621Z env: 2022-11-23T03:41:34.3078140Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:41:34.3078805Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:41:34.3079843Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:41:34.3081014Z CONTAINER_NAME: 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:41:34.3081755Z ##[endgroup] 2022-11-23T03:41:34.3834910Z With the provided path, there will be 1 file uploaded 2022-11-23T03:41:34.3836465Z Starting artifact upload 2022-11-23T03:41:34.3838672Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2022-11-23T03:41:34.3840239Z Artifact name is valid! 2022-11-23T03:41:34.5114177Z Container for artifact "usage-log-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437853.zip" successfully created. Starting upload of file(s) 2022-11-23T03:41:35.2363158Z Total size of all the files uploaded is 428216 bytes 2022-11-23T03:41:35.2364205Z File upload process has finished. Finalizing the artifact upload 2022-11-23T03:41:35.3387550Z Artifact has been finalized. All files have been successfully uploaded! 2022-11-23T03:41:35.3388245Z 2022-11-23T03:41:35.3388742Z The raw size of all the files that were specified for upload is 8883666 bytes 2022-11-23T03:41:35.3390102Z The size of all the files that were uploaded is 428216 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2022-11-23T03:41:35.3390971Z 2022-11-23T03:41:35.3393700Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2022-11-23T03:41:35.3395137Z 2022-11-23T03:41:35.3397414Z Artifact usage-log-runattempt1-test-distributed-2-2-linux.rocm.gpu_9655437853.zip has been successfully uploaded! 2022-11-23T03:41:35.3519805Z ##[group]Run set -x 2022-11-23T03:41:35.3520721Z set -x 2022-11-23T03:41:35.3521671Z python3 -m pip install -r requirements.txt 2022-11-23T03:41:35.3522675Z python3 -m pip install boto3==1.19.12 2022-11-23T03:41:35.3523612Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-11-23T03:41:35.3562305Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T03:41:35.3563043Z env: 2022-11-23T03:41:35.3563914Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:41:35.3564856Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:41:35.3566395Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:41:35.3567790Z CONTAINER_NAME: 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:41:35.3568858Z AWS_DEFAULT_REGION: us-east-1 2022-11-23T03:41:35.3569522Z BRANCH: master 2022-11-23T03:41:35.3570688Z TEST_CONFIG: distributed 2022-11-23T03:41:35.3571492Z SHARD_NUMBER: 2 2022-11-23T03:41:35.3572129Z BUILD_ENVIRONMENT: linux-focal-rocm5.2-py3.8 2022-11-23T03:41:35.3572715Z PR_NUMBER: 2022-11-23T03:41:35.3573254Z PYTORCH_RETRY_TEST_CASES: 1 2022-11-23T03:41:35.3573863Z PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1 2022-11-23T03:41:35.3574517Z SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T03:41:35.3575073Z TAG: 2022-11-23T03:41:35.3575523Z WORKFLOW_ID: 3528394938 2022-11-23T03:41:35.3614851Z GITHUB_TOKEN: *** 2022-11-23T03:41:35.3615480Z AWS_ACCESS_KEY_ID: *** 2022-11-23T03:41:35.3616187Z AWS_SECRET_ACCESS_KEY: *** 2022-11-23T03:41:35.3616686Z GHA_WORKFLOW_JOB_ID: 9655437853 2022-11-23T03:41:35.3617151Z ##[endgroup] 2022-11-23T03:41:35.3669424Z + python3 -m pip install -r requirements.txt 2022-11-23T03:41:36.2791426Z Collecting astunparse (from -r requirements.txt (line 2)) 2022-11-23T03:41:36.3461132Z Using cached https://files.pythonhosted.org/packages/2b/03/13dde6512ad7b4557eb792fbcf0c653af6076b81e5941d36ec61f7ce6028/astunparse-1.6.3-py2.py3-none-any.whl 2022-11-23T03:41:36.3513230Z Collecting expecttest (from -r requirements.txt (line 3)) 2022-11-23T03:41:36.3870089Z Using cached https://files.pythonhosted.org/packages/a6/26/1a287e44618c14659db0256bc1ee239c2134f9c863cb9a85813ecab73413/expecttest-0.1.4-py3-none-any.whl 2022-11-23T03:41:36.3889582Z Collecting future (from -r requirements.txt (line 4)) 2022-11-23T03:41:36.4493519Z Collecting hypothesis (from -r requirements.txt (line 5)) 2022-11-23T03:41:37.3056981Z Using cached https://files.pythonhosted.org/packages/82/7e/01691560a3a98bb1ae909affde49392f599eef3a0b91c27b992bbbde2abb/hypothesis-6.31.6-py3-none-any.whl 2022-11-23T03:41:37.3652783Z Collecting numpy (from -r requirements.txt (line 6)) 2022-11-23T03:41:38.1593358Z Using cached https://files.pythonhosted.org/packages/45/b2/6c7545bb7a38754d63048c7696804a0d947328125d81bf12beaa692c3ae3/numpy-1.19.5-cp36-cp36m-manylinux1_x86_64.whl 2022-11-23T03:41:38.4679085Z Collecting psutil (from -r requirements.txt (line 7)) 2022-11-23T03:41:38.7535253Z Collecting pyyaml (from -r requirements.txt (line 8)) 2022-11-23T03:41:38.8719028Z Using cached https://files.pythonhosted.org/packages/b3/85/79b9e5b4e8d3c0ac657f4e8617713cca8408f6cdc65d2ee6554217cedff1/PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl 2022-11-23T03:41:38.8869162Z Collecting requests (from -r requirements.txt (line 9)) 2022-11-23T03:41:38.9540295Z Using cached https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl 2022-11-23T03:41:38.9706319Z Collecting setuptools (from -r requirements.txt (line 10)) 2022-11-23T03:41:39.4087679Z Using cached https://files.pythonhosted.org/packages/b0/3a/88b210db68e56854d0bcf4b38e165e03be377e13907746f825790f3df5bf/setuptools-59.6.0-py3-none-any.whl 2022-11-23T03:41:39.4641408Z Collecting six (from -r requirements.txt (line 11)) 2022-11-23T03:41:39.4972620Z Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2022-11-23T03:41:39.4993677Z Collecting types-dataclasses (from -r requirements.txt (line 12)) 2022-11-23T03:41:39.5344428Z Using cached https://files.pythonhosted.org/packages/31/85/23ab2bbc280266af5bf22ded4e070946d1694d1721ced90666b649eaa795/types_dataclasses-0.6.6-py3-none-any.whl 2022-11-23T03:41:39.5360690Z Collecting typing_extensions (from -r requirements.txt (line 13)) 2022-11-23T03:41:39.5676689Z Using cached https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl 2022-11-23T03:41:39.5699441Z Collecting sympy (from -r requirements.txt (line 14)) 2022-11-23T03:41:39.6763132Z Using cached https://files.pythonhosted.org/packages/78/43/33c5a5e7fbafbf51520f4e09cb0634a1ca1d4cd5469c57967e43183d7a42/sympy-1.9-py3-none-any.whl 2022-11-23T03:41:39.9520366Z Collecting filelock (from -r requirements.txt (line 15)) 2022-11-23T03:41:39.9837445Z Using cached https://files.pythonhosted.org/packages/84/ce/8916d10ef537f3f3b046843255f9799504aa41862bfa87844b9bdc5361cd/filelock-3.4.1-py3-none-any.whl 2022-11-23T03:41:39.9939162Z Collecting networkx (from -r requirements.txt (line 16)) 2022-11-23T03:41:40.0945121Z Using cached https://files.pythonhosted.org/packages/f3/b7/c7f488101c0bb5e4178f3cde416004280fd40262433496830de8a8c21613/networkx-2.5.1-py3-none-any.whl 2022-11-23T03:41:40.1980528Z Collecting jinja2 (from -r requirements.txt (line 17)) 2022-11-23T03:41:40.2413021Z Using cached https://files.pythonhosted.org/packages/20/9a/e5d9ec41927401e41aea8af6d16e78b5e612bca4699d417f646a9610a076/Jinja2-3.0.3-py3-none-any.whl 2022-11-23T03:41:40.2502008Z Collecting wheel<1.0,>=0.23.0 (from astunparse->-r requirements.txt (line 2)) 2022-11-23T03:41:40.3125096Z Using cached https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl 2022-11-23T03:41:40.3186032Z Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis->-r requirements.txt (line 5)) 2022-11-23T03:41:40.3552612Z Using cached https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl 2022-11-23T03:41:40.3584951Z Collecting attrs>=19.2.0 (from hypothesis->-r requirements.txt (line 5)) 2022-11-23T03:41:40.3941589Z Using cached https://files.pythonhosted.org/packages/f2/bc/d817287d1aa01878af07c19505fafd1165cd6a119e9d0821ca1d1c20312d/attrs-22.1.0-py2.py3-none-any.whl 2022-11-23T03:41:40.4276736Z Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests->-r requirements.txt (line 9)) 2022-11-23T03:41:40.5481002Z Using cached https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl 2022-11-23T03:41:40.5531088Z Collecting idna<4,>=2.5; python_version >= "3" (from requests->-r requirements.txt (line 9)) 2022-11-23T03:41:40.5735738Z Using cached https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl 2022-11-23T03:41:40.5774433Z Collecting certifi>=2017.4.17 (from requests->-r requirements.txt (line 9)) 2022-11-23T03:41:40.6109610Z Using cached https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl 2022-11-23T03:41:40.6158664Z Collecting urllib3<1.27,>=1.21.1 (from requests->-r requirements.txt (line 9)) 2022-11-23T03:41:40.6754951Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T03:41:40.7007322Z Collecting mpmath>=0.19 (from sympy->-r requirements.txt (line 14)) 2022-11-23T03:41:40.7337144Z Using cached https://files.pythonhosted.org/packages/d4/cf/3965bddbb4f1a61c49aacae0e78fd1fe36b5dc36c797b31f30cf07dcbbb7/mpmath-1.2.1-py3-none-any.whl 2022-11-23T03:41:40.7590498Z Collecting decorator<5,>=4.3 (from networkx->-r requirements.txt (line 16)) 2022-11-23T03:41:40.7948295Z Using cached https://files.pythonhosted.org/packages/ed/1b/72a1821152d07cf1d8b6fce298aeb06a7eb90f4d6d41acec9861e7cc6df0/decorator-4.4.2-py2.py3-none-any.whl 2022-11-23T03:41:40.7969310Z Collecting MarkupSafe>=2.0 (from jinja2->-r requirements.txt (line 17)) 2022-11-23T03:41:40.9080342Z Using cached https://files.pythonhosted.org/packages/fc/d6/57f9a97e56447a1e340f8574836d3b636e2c14de304943836bd645fa9c7e/MarkupSafe-2.0.1-cp36-cp36m-manylinux1_x86_64.whl 2022-11-23T03:41:40.9115324Z Installing collected packages: six, wheel, astunparse, expecttest, future, sortedcontainers, attrs, hypothesis, numpy, psutil, pyyaml, charset-normalizer, idna, certifi, urllib3, requests, setuptools, types-dataclasses, typing-extensions, mpmath, sympy, filelock, decorator, networkx, MarkupSafe, jinja2 2022-11-23T03:41:50.6125859Z Successfully installed MarkupSafe-2.0.1 astunparse-1.6.3 attrs-22.1.0 certifi-2022.9.24 charset-normalizer-2.0.12 decorator-4.4.2 expecttest-0.1.4 filelock-3.4.1 future-0.18.2 hypothesis-6.31.6 idna-3.4 jinja2-3.0.3 mpmath-1.2.1 networkx-2.5.1 numpy-1.19.5 psutil-5.9.4 pyyaml-6.0 requests-2.27.1 setuptools-59.6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.9 types-dataclasses-0.6.6 typing-extensions-4.1.1 urllib3-1.26.12 wheel-0.37.1 2022-11-23T03:41:50.7862243Z + python3 -m pip install boto3==1.19.12 2022-11-23T03:41:51.6898943Z Collecting boto3==1.19.12 2022-11-23T03:41:52.4417644Z Using cached https://files.pythonhosted.org/packages/5e/e1/156846b09fca21b9b164c54200011e3bd17f29187cbfc6903a8e0281a304/boto3-1.19.12-py3-none-any.whl 2022-11-23T03:41:52.4571223Z Collecting s3transfer<0.6.0,>=0.5.0 (from boto3==1.19.12) 2022-11-23T03:41:52.4918721Z Using cached https://files.pythonhosted.org/packages/7b/9c/f51775ebe7df5a7aa4e7c79ed671bde94e154bd968aca8d65bb24aba0c8c/s3transfer-0.5.2-py3-none-any.whl 2022-11-23T03:41:52.4986423Z Collecting jmespath<1.0.0,>=0.7.1 (from boto3==1.19.12) 2022-11-23T03:41:52.5296376Z Using cached https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl 2022-11-23T03:41:52.5333006Z Collecting botocore<1.23.0,>=1.22.12 (from boto3==1.19.12) 2022-11-23T03:41:53.5504420Z Using cached https://files.pythonhosted.org/packages/6a/73/552b27e3a1b4f83630907c4958be78e9d4c906e73efd554ebd5e21cb1692/botocore-1.22.12-py3-none-any.whl 2022-11-23T03:41:53.8534399Z Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T03:41:53.8858421Z Using cached https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl 2022-11-23T03:41:53.8936868Z Collecting urllib3<1.27,>=1.25.4 (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T03:41:53.9516196Z Using cached https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl 2022-11-23T03:41:53.9753890Z Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) 2022-11-23T03:41:53.9959223Z Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl 2022-11-23T03:41:53.9982403Z Installing collected packages: six, python-dateutil, jmespath, urllib3, botocore, s3transfer, boto3 2022-11-23T03:41:54.5906794Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.12 2022-11-23T03:41:54.6775719Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-11-23T03:42:04.2662522Z [scribe] Scribe access token not provided, sending report via boto3... 2022-11-23T03:42:04.2663240Z 2022-11-23T03:42:04.2670076Z ----- Historic stats comparison result ------ 2022-11-23T03:42:04.2670817Z 2022-11-23T03:42:04.2671418Z job: linux-focal-rocm5.2-py3.8 2022-11-23T03:42:04.2672252Z commit: 1cfd3858ac54fe3883534309081631a0a892ba3f 2022-11-23T03:42:04.2672746Z 2022-11-23T03:42:04.2673274Z Commit graph (base is most recent master ancestor with at least one S3 report): 2022-11-23T03:42:04.2674020Z 2022-11-23T03:42:04.2674301Z : (master) 2022-11-23T03:42:04.2674970Z | 2022-11-23T03:42:04.2675806Z * 1cfd3858ac (HEAD) total time 3513.36s 2022-11-23T03:42:04.2682791Z * 26322544b8 (base) 2 reports, total time 10567.84s � 1421.60s 2022-11-23T03:42:04.2684284Z * 7f4b4d2827 2 reports, total time 8461.32s � 1168.97s 2022-11-23T03:42:04.2685535Z * b50699f247 2 reports, total time 9450.96s � 2404.87s 2022-11-23T03:42:04.2686812Z * 8bf8e4d71e 2 reports, total time 9492.36s � 2525.06s 2022-11-23T03:42:04.2688082Z * ce856cee7e 2 reports, total time 10405.71s � 1187.54s 2022-11-23T03:42:04.2689318Z * 391b593ca2 2 reports, total time 10409.50s � 1113.28s 2022-11-23T03:42:04.2690726Z * 5bba783d21 2 reports, total time 10442.70s � 1177.08s 2022-11-23T03:42:04.2691911Z * ea920a1115 2 reports, total time 8384.52s � 1109.09s 2022-11-23T03:42:04.2693050Z * 74e62a1fef 2 reports, total time 8516.95s � 1120.62s 2022-11-23T03:42:04.2694194Z * 00b7d8ef23 2 reports, total time 10408.58s � 1157.84s 2022-11-23T03:42:04.2694879Z | 2022-11-23T03:42:04.2695384Z : 2022-11-23T03:42:04.2695733Z 2022-11-23T03:42:04.2696145Z Removed (across 518 suites) 0 tests, totaling 0.00s 2022-11-23T03:42:04.2697042Z Modified (across 0 suites) 0 tests, totaling 0.00s 2022-11-23T03:42:04.2697942Z Added (across 86 suites) 930 tests, totaling +4101.48s 2022-11-23T03:42:04.3244582Z ##[group]Run # Only stop the docker container we started since there might be multiple runners on this host. 2022-11-23T03:42:04.3245972Z # Only stop the docker container we started since there might be multiple runners on this host. 2022-11-23T03:42:04.3247238Z docker stop "8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2" || true 2022-11-23T03:42:04.3248295Z # Prune all of the docker containers. 2022-11-23T03:42:04.3249265Z # Might fail if a prune is already in progress by another runner. 2022-11-23T03:42:04.3250197Z docker container prune -f || true 2022-11-23T03:42:04.3251261Z # Prune everything docker if there are more than 10 images (~200GB). 2022-11-23T03:42:04.3252290Z # This is easier than using a time filter, e.g., "until=24h". 2022-11-23T03:42:04.3253318Z # Might fail if a prune is already in progress by another runner. 2022-11-23T03:42:04.3254310Z image_count=$(docker images | wc -l) 2022-11-23T03:42:04.3255112Z if [[ ${image_count} -gt 10 ]]; then 2022-11-23T03:42:04.3255908Z  echo "Purging all docker caches" 2022-11-23T03:42:04.3256704Z  docker system prune -af || true 2022-11-23T03:42:04.3257393Z else 2022-11-23T03:42:04.3258214Z  echo "Will not purge docker, only ${image_count} images found" 2022-11-23T03:42:04.3259005Z fi 2022-11-23T03:42:04.3308462Z shell: /bin/bash --noprofile --norc -e -o pipefail {0} 2022-11-23T03:42:04.3309189Z env: 2022-11-23T03:42:04.3309787Z GIT_DEFAULT_BRANCH: master 2022-11-23T03:42:04.3310570Z DOCKER_HOST: unix:///run/user/1121/docker.sock 2022-11-23T03:42:04.3311806Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --group-add video --group-add daemon 2022-11-23T03:42:04.3313184Z CONTAINER_NAME: 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:42:04.3314055Z ##[endgroup] 2022-11-23T03:42:04.7593097Z 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:42:15.1762545Z Deleted Containers: 2022-11-23T03:42:15.1763841Z 8ba7ada97cfc204953085eabb66422cd11b1cfdef1f0afb05b664c64ca5461a2 2022-11-23T03:42:15.1764514Z 2022-11-23T03:42:15.1764845Z Total reclaimed space: 8.468GB 2022-11-23T03:42:15.2279619Z Will not purge docker, only 4 images found 2022-11-23T03:42:15.2356397Z Post job cleanup. 2022-11-23T03:42:15.2392209Z Post job cleanup. 2022-11-23T03:42:15.3703490Z [command]/usr/bin/git version 2022-11-23T03:42:15.3761087Z git version 2.35.1 2022-11-23T03:42:15.3817150Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/b9daf62b-b487-403e-9fa5-ac6b41cd7771' before making global git config changes 2022-11-23T03:42:15.3819291Z Adding repository directory to the temporary git global config as a safe directory 2022-11-23T03:42:15.3823152Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2022-11-23T03:42:15.3866217Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-11-23T03:42:15.3931836Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-11-23T03:42:15.4279004Z Entering 'android/libs/fbjni' 2022-11-23T03:42:15.4334591Z Entering 'third_party/FP16' 2022-11-23T03:42:15.4401742Z Entering 'third_party/FXdiv' 2022-11-23T03:42:15.4459733Z Entering 'third_party/NNPACK' 2022-11-23T03:42:15.4529204Z Entering 'third_party/QNNPACK' 2022-11-23T03:42:15.4597644Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T03:42:15.4660770Z Entering 'third_party/XNNPACK' 2022-11-23T03:42:15.4735447Z Entering 'third_party/benchmark' 2022-11-23T03:42:15.4793024Z Entering 'third_party/cpuinfo' 2022-11-23T03:42:15.4862930Z Entering 'third_party/cub' 2022-11-23T03:42:15.4933974Z Entering 'third_party/cudnn_frontend' 2022-11-23T03:42:15.5014561Z Entering 'third_party/cutlass' 2022-11-23T03:42:15.5092857Z Entering 'third_party/eigen' 2022-11-23T03:42:15.5152049Z Entering 'third_party/fbgemm' 2022-11-23T03:42:15.5223140Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T03:42:15.5293600Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T03:42:15.5361496Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T03:42:15.5431118Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T03:42:15.5495282Z Entering 'third_party/flatbuffers' 2022-11-23T03:42:15.5566454Z Entering 'third_party/fmt' 2022-11-23T03:42:15.5638661Z Entering 'third_party/foxi' 2022-11-23T03:42:15.5704689Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T03:42:15.5771656Z Entering 'third_party/gloo' 2022-11-23T03:42:15.5841144Z Entering 'third_party/googletest' 2022-11-23T03:42:15.5913066Z Entering 'third_party/ideep' 2022-11-23T03:42:15.5982181Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T03:42:15.6053450Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T03:42:15.6136477Z Entering 'third_party/ios-cmake' 2022-11-23T03:42:15.6200209Z Entering 'third_party/ittapi' 2022-11-23T03:42:15.6270681Z Entering 'third_party/kineto' 2022-11-23T03:42:15.6332216Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T03:42:15.6400326Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T03:42:15.6456683Z Entering 'third_party/nccl/nccl' 2022-11-23T03:42:15.6513181Z Entering 'third_party/neon2sse' 2022-11-23T03:42:15.6577053Z Entering 'third_party/nlohmann' 2022-11-23T03:42:15.6645850Z Entering 'third_party/onnx' 2022-11-23T03:42:15.6735615Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T03:42:15.6798045Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T03:42:15.6872265Z Entering 'third_party/onnx-tensorrt' 2022-11-23T03:42:15.6940647Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T03:42:15.7012332Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T03:42:15.7077350Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T03:42:15.7146408Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T03:42:15.7232468Z Entering 'third_party/pocketfft' 2022-11-23T03:42:15.7298096Z Entering 'third_party/protobuf' 2022-11-23T03:42:15.7373260Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T03:42:15.7434179Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T03:42:15.7506494Z Entering 'third_party/psimd' 2022-11-23T03:42:15.7574661Z Entering 'third_party/pthreadpool' 2022-11-23T03:42:15.7643400Z Entering 'third_party/pybind11' 2022-11-23T03:42:15.7712293Z Entering 'third_party/python-enum' 2022-11-23T03:42:15.7782622Z Entering 'third_party/python-peachpy' 2022-11-23T03:42:15.7850149Z Entering 'third_party/python-six' 2022-11-23T03:42:15.7919118Z Entering 'third_party/sleef' 2022-11-23T03:42:15.7988050Z Entering 'third_party/tbb' 2022-11-23T03:42:15.8054890Z Entering 'third_party/tensorpipe' 2022-11-23T03:42:15.8117373Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T03:42:15.8186811Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T03:42:15.8250780Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T03:42:15.8314466Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T03:42:15.8383659Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T03:42:15.8450095Z Entering 'third_party/zstd' 2022-11-23T03:42:15.8538057Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-11-23T03:42:15.8595030Z http.https://github.com/.extraheader 2022-11-23T03:42:15.8615416Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2022-11-23T03:42:15.8685630Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-11-23T03:42:15.9085857Z Entering 'android/libs/fbjni' 2022-11-23T03:42:15.9122788Z http.https://github.com/.extraheader 2022-11-23T03:42:15.9177311Z Entering 'third_party/FP16' 2022-11-23T03:42:15.9212891Z http.https://github.com/.extraheader 2022-11-23T03:42:15.9276804Z Entering 'third_party/FXdiv' 2022-11-23T03:42:15.9312025Z http.https://github.com/.extraheader 2022-11-23T03:42:15.9368336Z Entering 'third_party/NNPACK' 2022-11-23T03:42:15.9402042Z http.https://github.com/.extraheader 2022-11-23T03:42:15.9458676Z Entering 'third_party/QNNPACK' 2022-11-23T03:42:15.9494258Z http.https://github.com/.extraheader 2022-11-23T03:42:15.9556999Z Entering 'third_party/VulkanMemoryAllocator' 2022-11-23T03:42:15.9592796Z http.https://github.com/.extraheader 2022-11-23T03:42:15.9647480Z Entering 'third_party/XNNPACK' 2022-11-23T03:42:15.9681449Z http.https://github.com/.extraheader 2022-11-23T03:42:15.9755103Z Entering 'third_party/benchmark' 2022-11-23T03:42:15.9791965Z http.https://github.com/.extraheader 2022-11-23T03:42:15.9853311Z Entering 'third_party/cpuinfo' 2022-11-23T03:42:15.9890202Z http.https://github.com/.extraheader 2022-11-23T03:42:15.9951216Z Entering 'third_party/cub' 2022-11-23T03:42:15.9990002Z http.https://github.com/.extraheader 2022-11-23T03:42:16.0041638Z Entering 'third_party/cudnn_frontend' 2022-11-23T03:42:16.0077728Z http.https://github.com/.extraheader 2022-11-23T03:42:16.0146559Z Entering 'third_party/cutlass' 2022-11-23T03:42:16.0181434Z http.https://github.com/.extraheader 2022-11-23T03:42:16.0249350Z Entering 'third_party/eigen' 2022-11-23T03:42:16.0285377Z http.https://github.com/.extraheader 2022-11-23T03:42:16.0352987Z Entering 'third_party/fbgemm' 2022-11-23T03:42:16.0387702Z http.https://github.com/.extraheader 2022-11-23T03:42:16.0448010Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-11-23T03:42:16.0484687Z http.https://github.com/.extraheader 2022-11-23T03:42:16.0545522Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-11-23T03:42:16.0580318Z http.https://github.com/.extraheader 2022-11-23T03:42:16.0636849Z Entering 'third_party/fbgemm/third_party/googletest' 2022-11-23T03:42:16.0671934Z http.https://github.com/.extraheader 2022-11-23T03:42:16.0733692Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2022-11-23T03:42:16.0767474Z http.https://github.com/.extraheader 2022-11-23T03:42:16.0832324Z Entering 'third_party/flatbuffers' 2022-11-23T03:42:16.0868888Z http.https://github.com/.extraheader 2022-11-23T03:42:16.0933589Z Entering 'third_party/fmt' 2022-11-23T03:42:16.0968554Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1029969Z Entering 'third_party/foxi' 2022-11-23T03:42:16.1065421Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1126077Z Entering 'third_party/gemmlowp/gemmlowp' 2022-11-23T03:42:16.1160020Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1214699Z Entering 'third_party/gloo' 2022-11-23T03:42:16.1244460Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1303858Z Entering 'third_party/googletest' 2022-11-23T03:42:16.1339819Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1397481Z Entering 'third_party/ideep' 2022-11-23T03:42:16.1434450Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1484603Z Entering 'third_party/ideep/mkl-dnn' 2022-11-23T03:42:16.1517987Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1572733Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-11-23T03:42:16.1608226Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1685346Z Entering 'third_party/ios-cmake' 2022-11-23T03:42:16.1716177Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1777861Z Entering 'third_party/ittapi' 2022-11-23T03:42:16.1814274Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1868775Z Entering 'third_party/kineto' 2022-11-23T03:42:16.1894389Z http.https://github.com/.extraheader 2022-11-23T03:42:16.1951387Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-11-23T03:42:16.1988448Z http.https://github.com/.extraheader 2022-11-23T03:42:16.2045819Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-11-23T03:42:16.2081223Z http.https://github.com/.extraheader 2022-11-23T03:42:16.2148575Z Entering 'third_party/nccl/nccl' 2022-11-23T03:42:16.2183007Z http.https://github.com/.extraheader 2022-11-23T03:42:16.2241769Z Entering 'third_party/neon2sse' 2022-11-23T03:42:16.2275465Z http.https://github.com/.extraheader 2022-11-23T03:42:16.2334264Z Entering 'third_party/nlohmann' 2022-11-23T03:42:16.2373514Z http.https://github.com/.extraheader 2022-11-23T03:42:16.2417819Z Entering 'third_party/onnx' 2022-11-23T03:42:16.2449161Z http.https://github.com/.extraheader 2022-11-23T03:42:16.2534968Z Entering 'third_party/onnx/third_party/benchmark' 2022-11-23T03:42:16.2570086Z http.https://github.com/.extraheader 2022-11-23T03:42:16.2635439Z Entering 'third_party/onnx/third_party/pybind11' 2022-11-23T03:42:16.2669117Z http.https://github.com/.extraheader 2022-11-23T03:42:16.2731053Z Entering 'third_party/onnx-tensorrt' 2022-11-23T03:42:16.2767116Z http.https://github.com/.extraheader 2022-11-23T03:42:16.2819650Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-11-23T03:42:16.2849054Z http.https://github.com/.extraheader 2022-11-23T03:42:16.2913348Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-11-23T03:42:16.2948630Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3010147Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-11-23T03:42:16.3045989Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3109817Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-11-23T03:42:16.3146839Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3210096Z Entering 'third_party/pocketfft' 2022-11-23T03:42:16.3246133Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3303831Z Entering 'third_party/protobuf' 2022-11-23T03:42:16.3335993Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3393380Z Entering 'third_party/protobuf/third_party/benchmark' 2022-11-23T03:42:16.3430027Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3488670Z Entering 'third_party/protobuf/third_party/googletest' 2022-11-23T03:42:16.3524367Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3574543Z Entering 'third_party/psimd' 2022-11-23T03:42:16.3611494Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3674876Z Entering 'third_party/pthreadpool' 2022-11-23T03:42:16.3710716Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3778217Z Entering 'third_party/pybind11' 2022-11-23T03:42:16.3814330Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3870373Z Entering 'third_party/python-enum' 2022-11-23T03:42:16.3907645Z http.https://github.com/.extraheader 2022-11-23T03:42:16.3968433Z Entering 'third_party/python-peachpy' 2022-11-23T03:42:16.4003997Z http.https://github.com/.extraheader 2022-11-23T03:42:16.4064826Z Entering 'third_party/python-six' 2022-11-23T03:42:16.4096995Z http.https://github.com/.extraheader 2022-11-23T03:42:16.4156001Z Entering 'third_party/sleef' 2022-11-23T03:42:16.4191281Z http.https://github.com/.extraheader 2022-11-23T03:42:16.4253449Z Entering 'third_party/tbb' 2022-11-23T03:42:16.4288087Z http.https://github.com/.extraheader 2022-11-23T03:42:16.4353025Z Entering 'third_party/tensorpipe' 2022-11-23T03:42:16.4383270Z http.https://github.com/.extraheader 2022-11-23T03:42:16.4442525Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-11-23T03:42:16.4478813Z http.https://github.com/.extraheader 2022-11-23T03:42:16.4533477Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-11-23T03:42:16.4558894Z http.https://github.com/.extraheader 2022-11-23T03:42:16.4623695Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-11-23T03:42:16.4656911Z http.https://github.com/.extraheader 2022-11-23T03:42:16.4717047Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-11-23T03:42:16.4747854Z http.https://github.com/.extraheader 2022-11-23T03:42:16.4804141Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-11-23T03:42:16.4836833Z http.https://github.com/.extraheader 2022-11-23T03:42:16.4894601Z Entering 'third_party/zstd' 2022-11-23T03:42:16.4930465Z http.https://github.com/.extraheader 2022-11-23T03:42:16.5414909Z Cleaning up orphan processes